title: Converting a series of matrix files into an index of coordinates in awk
body: <p>I have a time series of files 0000.vx.dat, 0000.vy.dat, 0000.vz.dat; ...; 0077.vx.dat, 0077.vy.dat, 0077.vz.dat... Each file is a space-separated 2D matrix. I would like to take each triplet of files and combine them all into a coordinate-based data format, i.e.:</p>
  
  <p>[timestep + 1] [i] [j] [vx(i,j)] [vy(i,j)] [vz(i,j)]</p>
  
  <p>Each file number corresponds to a particular time step. Given the amount of data I have in this time series (~ 4 GB), bash wasn't cutting it so it seemed to be time to head over to awk... specifically mawk. It was pretty stupid to try this in bash but here is 
  my ill-fated attempt:</p>
  
  <pre><code>for x in $(seq 1 78)
  do
    tfx=${tf[$x]} # an array of padded zeros
    for y in $(seq 1 1568)
    do
      for z in $(seq 1 1344)
      do
        echo $x $y $z $(awk -v i=$z -v j=$y "FNR == i {print j}" $tfx.vx.dat) $(awk -v i=$z -v j=$y "FNR == i {print j}" $tfx.vy.dat) $(awk -v i=$z -v j=$y "FNR == i {print j}" $tfx.vz.dat) &gt;&gt; $file
      done
    done
  done
  </code></pre>
  
  <p>edit: Thank you, ruakh, for pointing out that I had kept j in shell variable format with a $ in front! This is just a snippet of the original script, but I guess would be considered the guts of it!</p>
  
  <p>Suffice it to say this would have taken about six months because of all the memory overhead in bash associated with O(MxN) algorithms, subshells and pipes and whatnot. I was looking for more along the lines of a day at most. Each file is around 18 MB, so it should not be that much of a problem. I would be happy with doing this one timestep at a time in awk provided that I get one output file per timestep. I could just cat them all together without much issue afterwords, I think. It is important, though, that the time step number be the first item on the coordinate list. I could achieve this with an awk -v argument (see above) in with a bash routine. I do not know how to look up specific elements of matrices in three separate files and put them all together into one output. This is the main hurdle I would like to overcome. I was hoping mawk could provide a nice balance between effort and computational speed. If this seems to be too much for an awk script, I could go to something lower level, and would appreciate any of those answering letting me know I should just go to C instead.</p>
  
  <p>Thank you in advance! I really like awk, but am afraid I am a novice.</p>
  
  <p>The three files, 0000.vx.dat, 0000.vy.dat, and 0000.vz.dat would read as follows (except huge and of the correct dimensions):</p>
  
  <p>0000.vx.dat:</p>
  
  <pre><code>1 2 3
  4 5 6
  7 8 9
  </code></pre>
  
  <p>0000.vy.dat:</p>
  
  <pre><code>10 11 12
  13 14 15
  16 17 18
  </code></pre>
  
  <p>0000.vz.dat:</p>
  
  <pre><code>19 20 21
  22 23 24
  25 26 27
  </code></pre>
  
  <p>I would like to be able to input:</p>
  
  <pre><code>awk -v t=1 -f stackoverflow.awk 0000.vx.dat 0000.vy.dat 0000.vz.dat
  </code></pre>
  
  <p>and get the following output:</p>
  
  <pre><code>1 1 1 1 10 19
  1 1 2 2 11 20
  1 1 3 3 12 21
  1 2 1 4 13 22
  1 2 2 5 14 23
  1 2 3 6 15 24
  1 3 1 7 16 25
  1 3 2 8 17 26
  1 3 3 9 18 27
  </code></pre>
  
  <p>edit: Thank you, shellter, for suggesting I put the desired input and output more clearly!</p>