title: Failure to write output of shell command to file in Go
body: <p>I have written the following function for executing the <code>snt2cooc</code> command (one of the preprocessing steps for running <a href="http://www.statmt.org/moses/giza/GIZA++.html" rel="nofollow">GIZA++</a>. For our purposes I think we can consider the <code>snt2cooc</code> script to be a black box):</p>
  
  <pre><code>func SNTToCOOC(srcVocab, tgtVocab, sntPath, outpath string) error {
      // open the out file for writing
      outfile, err := os.Create(outpath)
      if err != nil {
          return err
      }
      defer outfile.Close()
  
      cmdStr := "snt2cooc"
      args := []string{srcVocab, tgtVocab, sntPath}
      cmd := exec.Command(cmdStr, args...)
      cmd.Stdout = outfile
      if err = cmd.Run(); err != nil {
          return err
      }
      cmd.Wait()
      return err
  }
  </code></pre>
  
  <p>When running, the function executes without an error, but the output file is empty. This same code works for other similar commands, but not for this specific <code>snt2cooc</code> command, and I noticed that when I run this command in the shell directly: </p>
  
  <p>snt2cooc file1.vcb file2.vcb file3.snt</p>
  
  <p>I get the following output:</p>
  
  <pre><code>END.
  0 2
  0 3
  0 4
  0 5
  0 6
  </code></pre>
  
  <p>(truncated for brevity)</p>
  
  <p>And if I send the output of the command to a file directly from the shell:</p>
  
  <pre><code>snt2cooc file1.vcb file2.vcb file3.snt &gt; out.txt
  </code></pre>
  
  <p>The contents of <code>out.txt</code> are as expected:</p>
  
  <pre><code>0 2
  0 3
  0 4
  0 5
  0 6
  </code></pre>
  
  <p>Notice how in the first case, the line <code>END.</code> is output to stdout first, and only then is the real output of the command sent to stdout. I therefore think there is a race condition going on, where the Go code finishes executing before the command's final output is written to file. This despite calling <code>cmd.Wait()</code>. I'm not too sure what exactly the <code>snt2cooc</code> command is doing internally. Could someone provide a hint on how to solve this?</p>
  
  <p><strong>Edit 1:</strong></p>
  
  <p>It seems like the following code, with the sleep of 500ms included, consistently writes output to the file for the <code>snt2cooc</code> command:</p>
  
  <pre><code>cmdStr := "snt2cooc"
  args := []string{srcVocab, tgtVocab, sntPath}
  cmd := exec.Command(cmdStr, args...)
  stdout, err := cmd.StdoutPipe()
  time.Sleep(500 * time.Millisecond)
  if err != nil {
      return err
  }
  err = cmd.Start()
  if err != nil {
      return err
  }
  
  out := bufio.NewScanner(stdout)
  for out.Scan() {
      outfile.Write(out.Bytes())
      outfile.WriteString("\n")
  }
  if err := out.Err(); err != nil {
      return err
  }
  </code></pre>
  
  <p>This proves to me that there is some race condition going on, with the Go program exiting before all output is written to file. I added a bounty to this question, with the hope that someone can 1) explain why this is happening and 2) provide a non-hacky way (i.e. 500ms sleep) to fix it. </p>