title: Programmable arguments in perl pipes
body: <p>I'm gradually working my way up the perl learning curve (with thanks to contributors to this REALLY helpful site), but am struggling with how to approach this particular issue.</p>
  
  <p>I'm building a perl utility which utilises three (c++) third party programmes. Normally these are run: A $file_list | B -args | C $file_out</p>
  
  <p>where process A reads multiple files, process B modifies each individual file and process C collects all input files in the pipe and produces a single output file, with a null input file signifying the end of the input stream.</p>
  
  <p>The input files are large(ish) at around 100Mb and around 10 in number. The processes are CPU intensive and the whole process need to be applied to thousands of groups of files each day, so the simple solution of reading and writing intermediate files to disk is simply too inefficient. In addition, the process above is only part of a processing sequence, where the input files are already in memory and the output file also needs to be in memory for further processing.</p>
  
  <p>There are a number of solutions to this already well documented and I have a prototype version utilising  IPC::Open3(). So far, so good. :)</p>
  
  <p>However - when piping each file to process A through process B <strong>I need to modify the arguments in process B for each input file without interrupting the forward flow to process C.</strong> This is where I come unstuck and am looking for some suggestions.</p>
  
  <p>As further background:</p>
  
  <ol>
  <li>Running in Ubuntu 16.04 LTS (currently within Virtual box)and perl v5.22.1</li>
  <li>The programme will run on (and within) a single machine by one user (me !), i.e. no external network communication or multi user or public requirement - so simplicity of programming is preferred over strong security.</li>
  <li>Since the process must run repeatedly without interruption, robust/reliable I/O handling is required.</li>
  <li>I have access to the source code of each process, so that could be modified (although I'd prefer not to).</li>
  </ol>
  
  <p>My apologies for the lack of "code to date", but I thought the question is more one of "How do I approach this?" rather than "How do I get my code to work?".</p>
  
  <p>Any pointers or help would be very much appreciated.</p>