title: python script using subprocess, redirect ALL output to file
body: <p>I am writing something for static analysis of source code in different languages. As anything has to be open source and callable from command line I now have downloaded one tool per language. So I decided to write a python script listing all source files in a project folder and calling the respective tool.</p>
  
  <p>So part of my code looks like this:</p>
  
  <pre><code>import os
  import sys
  import subprocess
  from subprocess import call
  from pylint.lint import Run as pylint
  
  
  class Analyser:
  
      def __init__(self, source=os.getcwd(), logfilename=None):
  
          # doing initialization stuff
          self.logfilename = logfilename or 'CodeAnalysisReport.log'
  
          self.listFiles()
          self.analyseFiles()
  
  
      def listFiles(self):
      # lists all source files in the specified directory
  
  
      def analyseFiles(self):
  
          self.analysePythons()
          self.analyseCpps()
          self.analyseJss()
          self.analyseJavas()
          self.analyseCs()
  
  
  if __name__ == '__main__':
  
      Analyser()
  </code></pre>
  
  <p>Let's have at a look at the C++ files part (I use Cppcheck to analyse those):</p>
  
  <pre><code>    def analyseCpps(self):
  
          for sourcefile in self.files['.cc'] + self.files['.cpp']:
              print '\n'*2, '*'*70, '\n', sourcefile
              call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile])
  </code></pre>
  
  <p>The console output for one of the files (it's just a random downloaded file) is:</p>
  
  <pre><code>**********************************************************************
  C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
  Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
  [C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
  [C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
  [C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
  (information) Cppcheck cannot find all the include files (use --check-config for details)
  </code></pre>
  
  <p>Line 1 and 2 coming from my script, lines 3 to 7 coming from Cppcheck.</p>
  
  <p>And this is what I want to save to my log file, for all the other files too. Everything in one single file.</p>
  
  <p>Of course I have searched SO and found some methods. But none is working completely.</p>
  
  <p><strong>First try:</strong></p>
  
  <p>Adding <code>sys.stdout = open(self.logfilename, 'w')</code> to my constructor. This makes line 1 and 2 of the above showed output be written to my log file. The rest is still shown on console.</p>
  
  <p><strong>Second try:</strong></p>
  
  <p>Additionaly, in <code>analyseCpps</code> I use:</p>
  
  <p><code>call(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], <strong>stdout=sys.stdout</strong>)</code></p>
  
  <p>This makes my log file to be:</p>
  
  <pre><code>Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
  
  
  ********************************************************************** 
  C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
  </code></pre>
  
  <p>and the console output is:</p>
  
  <pre><code>[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
  [C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
  [C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
  </code></pre>
  
  <p>Not what I want.</p>
  
  <p><strong>Third try:</strong></p>
  
  <p>Using <code>Popen</code> with <code>pipe</code>. <code>sys.stdout</code> is back to default.</p>
  
  <p>As preliminary work <code>analyseCpps</code> now is:</p>
  
  <pre><code>for sourcefile in self.files['.cc'] + self.files['.cpp']:
  
      print '\n'*2, '*'*70, '\n', sourcefile
      p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=subprocess.PIPE)
      p.stdout.read()
  </code></pre>
  
  <p><code>p.stdout.read()</code> shows only the last line of my desired output (line 7 in code box 3) </p>
  
  <p><strong>Fourth try:</strong></p>
  
  <p>Using <code>subprocess.Popen(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], <strong>stdout=open(self.logfilename, 'a+')</strong>)</code> just writes the one line <code>Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...</code> to my logfile, the rest is shown on the console.</p>
  
  <p><strong>Fifth try:</strong></p>
  
  <p>Instead of <code>subprocess.Popen</code> I use <code>os.system</code>, so my calling command is:</p>
  
  <p><code>os.system('C:\CodeAnalysis\cppcheck\cppcheck --enable=all %s >> %s' % (sourcefile, self.logfilename))</code></p>
  
  <p>This results in the same log file as my fourth try. If I type the same command directly in the windows console the result is the same. So I guess it it is not exactly a python problem but still: </p>
  
  <p><strong><em>If it is on the console there must be a way to put it in a file.</em></strong> Any ideas?</p>
  
  <hr>
  
  <p><strong><em>E D I T</em></strong></p>
  
  <p>Foolish me. I'm still a noob so I forgot about the <code>stderr</code>. That's where the decisive messages are going to. </p>
  
  <p>So now I have:</p>
  
  <pre><code>def analyseCpps(self):
  
      for sourcefile in self.files['.cc'] + self.files['.cpp']:
          p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stderr=subprocess.PIPE)
          with open(self.logfilename, 'a+') as logfile:
              logfile.write('%s\n%s\n' % ('*'*70, sourcefile))
              for line in p.stderr.readlines():
                  logfile.write('%s\n' % line.strip())
  </code></pre>
  
  <p>and it's working fine. </p>
  
  <hr>
  
  <p><strong><em>ANOTHER EDIT</em></strong></p>
  
  <p>according to Didier's answer:</p>
  
  <p>with <code>sys.stdout = open(self.logfilename, 'w', 0)</code> in my constructor:</p>
  
  <pre><code>def analyseCpps(self):
  
      for sourcefile in self.files['.cc'] + self.files['.cpp']:
          print '\n'*2, '*'*70, '\n', sourcefile
          p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout, stderr=sys.stdout)
  </code></pre>