title: python subprocess.Popen vs os.popen
body: <p>I'm trying to get the output of the following shell command in my python script,</p>
  
  <pre><code>hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}
  </code></pre>
  
  <p>I can successfully get the output through <code>os.popen</code> as follows:</p>
  
  <pre><code>import os
  cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
  p = os.popen(cmd,"r")
  while 1:
      line = p.readline()
      if not line: break
      print line
  </code></pre>
  
  <p>But <code>os.popen()</code> is deprecated since python 2.6 so I wanted to replace the above snippet with the <code>subprocess.Popen()</code> function.</p>
  
  <p>But the code snippet for <code>subprocess.Popen()</code> below gives a different result than the code snippet above.</p>
  
  <pre><code>import subprocess as sub
  import shlex
  cmd = "hadoop fs -ls /projectpath/ | grep ^d | grep -v done | head -1 | awk {'print $8'}"
  args = shlex.split(cmd)
  p = sub.Popen(args,stdout=sub.PIPE,stderr=sub.PIPE)
  output, errors = p.communicate()
  print output
  </code></pre>
  
  <p>The above command just gives output for <code>'hadoop fs -ls /projectpath/'</code> part of the command.
  I have tried consulting several references (<a href="http://docs.python.org/2/library/subprocess.html#popen-objects" rel="nofollow noreferrer">http://docs.python.org/2/library/subprocess.html#popen-objects</a>, <a href="https://stackoverflow.com/questions/3791465/python-os-system-for-command-line-call-linux-not-returning-what-it-should">Python, os.system for command-line call (linux) not returning what it should?</a>) for <code>subpocess.Popen()</code> but cannot get it to execute the command in the string cmd. Can anyone point out what I'm doing wrong?</p>