title: Add text on certain lines of a file, with the added text depending on the output of a command that takes a substring from the line
body: <p>I'm trying to make a shell script to take an input file (thousands of lines) and produce an output file that is the same except that on certain lines there will be added text.  When the text is added to the (middle of) the line, the exact added text will depend on a substring on the line.  The correlation between the substring and the added text is complex and comes from an external program that I can call in the shell.  I don't have the source for this converter program nor any control over how the mapping is done.</p>
  <p>To explain further...</p>
  <p>I have an input file of this general format:</p>
  <pre><code>Blah Blah Unimportant
  Something Something
  FIELD_INFO(field_name_1, output_1),
    FIELD_INFO(field_name_2, output_2),
  Yadda Yadda
  </code></pre>
  <p>The whole file needs to be copied, with added text, but the only important parts for me are the field names (e.g. field_name_1, field_name_2).  I have a command line program called &quot;converter&quot; that can take a file of field names and output a list of corresponding actions.  Converter cannot operate directly on the input file.  The input to converter needs to be <em>just</em> field names and the output of converter has extra information I don't need:</p>
  <pre><code>converter_field_name_1 &quot;action1&quot; /* Use this action for field_name_1 */
  converter_field_name_2 &quot;action2&quot; /* use this action for field_name_2 */
  </code></pre>
  <p>The desire is to create a second file that looks like this:</p>
  <pre><code>Blah Blah Unimportant
  Something Something
  FIELD_INFO(field_name_1, action1, output_1),
    FIELD_INFO(filed_name_2, action2, output_2),
  Yadda Yadda
  </code></pre>
  <p>Here is the script I'm working on, but I've hit a wall (or two):</p>
  <pre><code>#!/bin/bash
  
  filename=&quot;input_file&quot;
  
  # Let's create an array of the field names to feed to the converter program
  field_array=($(sed -e '/^\s*FIELD_INFO/ s/FIELD_INFO(\(.*\),.*),/\1/' -e 't' -e 'd' &lt; ${filename}))
  
  # Save the array to a file, to be able to use the converter's file option
  printf &quot;%s\n&quot; &quot;${field_array[@]}&quot; &gt; script_field_names.txt
  
  # Use converter on the whole file and extract only the actions into another array
  action_array=($(converter -f script_field_names.txt | cut -d'&quot;' -f 2))
  
  # I will make and use an associative array and try to use
  # sed to do the substitution
  declare -A mapper
  for i in ${!field_array[*]}
  do
  mapper[${field_array[i]}]=${action_array[i]}
  done
  
  #Now go back through the file and add action names (source file unchanged)
  sed -e &quot;s/FIELD_INFO(\(.*\),\(.*?),\)/FIELD_INFO(\1, ${mapper[\1], \2}/&quot; &lt; ${filename}
  </code></pre>
  <p>I know now that I can't use the sed group capture &quot;\1&quot; as an index into the mapper array like this. It is not working as a key and the output looks like this:</p>
  <pre><code>Blah Blah Unimportant
  Something Something
  FIELD_INFO(field_name_1, ,  output_1),
    FIELD_INFO(field_name_2, ,  output_2),
  Yadda Yadda
  
  </code></pre>
  <p>My actual script has debug statements scattered throughout and I know the field array, action array, and mapper array are all getting created correctly.  But my idea of using the group capture substring from sed as the index into the mapper array is not working because I now know that sed expands the variables <em>before</em> running in the sub-shell, so the mapper[] array is not seeing the substring as an index.</p>
  <p>What should I be doing instead?  This script may only be used once, but it's too time consuming and error prone to do the addition of the action strings by hand.  I want to come up with a way to make this work but I can't tell if I'm close or completely on the wrong path.</p>