title: How to run a bash script from within python and get all the output?
body: <p>This is a direct clarification question to the answer in <a href="https://stackoverflow.com/questions/59952928/how-to-execute-a-bash-script-from-within-python-in-a-non-blocking-way-and-allow/59954630">here</a> which I thought it worked, but it does not!</p>
  
  <p>I have the following test bash script (<code>testbash.sh</code>) which just creates some output and a lot of errors for testing purposes (running on Red Hat Enterprise Linux Server release 7.6 (Maipo) and also Ubuntu 16.04.6 LTS):</p>
  
  <pre><code>export MAX_SEED=2
  echo "Start test"
  pids=""
  
  for seed in `seq 1 ${MAX_SEED}`
  do
    python -c "raise ValueError('test')" &amp;
    pids="${pids} $!"
  done
  echo "pids: ${pids}"
  wait $pids
  echo "End test"
  </code></pre>
  
  <p>If I run this script I get the following output:</p>
  
  <pre><code>Start test
  pids:  68322 68323
  Traceback (most recent call last):
    File "&lt;string&gt;", line 1, in &lt;module&gt;
  ValueError: test
  Traceback (most recent call last):
    File "&lt;string&gt;", line 1, in &lt;module&gt;
  ValueError: test
  [1]-  Exit 1                  python -c "raise ValueError('test')"
  [2]+  Exit 1                  python -c "raise ValueError('test')"
  End test
  </code></pre>
  
  <p>That is the expected outcome. That is fine. I want to get errors!</p>
  
  <p>Now here is the python code that is supposed to catch all the output: </p>
  
  <pre><code>from __future__ import print_function
  
  import sys
  import time
  from subprocess import PIPE, Popen, STDOUT
  from threading  import Thread
  
  try:
      from queue import Queue, Empty
  except ImportError:
      from Queue import Queue, Empty  # python 2.x    
  
  ON_POSIX = 'posix' in sys.builtin_module_names
  
  def enqueue_output(out, queue):
      for line in iter(out.readline, b''):
          queue.put(line.decode('ascii'))
      out.close()
  
  p = Popen(['. testbash.sh'], stdout=PIPE, stderr=STDOUT, bufsize=1, close_fds=ON_POSIX, shell=True)
  q = Queue()
  t = Thread(target=enqueue_output, args=(p.stdout, q))
  t.daemon = True # thread dies with the program
  t.start()
  
  # read line without blocking
  while t.is_alive():
      #time.sleep(1)
      try:
          line = q.get(timeout=.1)
      except Empty:
          print(line)
          pass
      else:
          # got line
          print(line, end='')
  
  p.wait()
  print('returncode = {}'.format(p.returncode))
  </code></pre>
  
  <p>But when I run this code I only get the following output:</p>
  
  <pre><code>Start test
  pids:  70191 70192
  Traceback (most recent call last):
  returncode = 0
  </code></pre>
  
  <p>or this output (without the line <code>End test</code>):</p>
  
  <pre><code>Start test
  pids:  10180 10181
  Traceback (most recent call last):
    File "&lt;string&gt;", line 1, in &lt;module&gt;
  ValueError: test
  Traceback (most recent call last):
    File "&lt;string&gt;", line 1, in &lt;module&gt;
  ValueError: test
  returncode = 0
  </code></pre>
  
  <p>Most of the above output is missing! How can I fix this? Also, I need some way to check if any command in the bash script did not succeed. In the example this is the case, but the errorcode printed out is still 0. I expect an errorcode != 0.</p>
  
  <p>It is not important to immediately get the output. A delay of some seconds is fine. Also if the output order is a bit mixed up this is of no concern. The important thing is to get <em>all</em> the output (<code>stdout</code> and <code>stderr</code>).</p>
  
  <p>Maybe there is a simpler way to just get the output of a bash script which is started from python?</p>