title: Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory"
body: <p><strong>Note:</strong> This question was originally asked <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory">here</a> but the bounty time expired even though an acceptable answer was not actually found. I am re-asking this question including all details provided in the original question.</p>
  
  <p>A python script is running a set of class functions every 60 seconds using the <a href="http://docs.python.org/library/sched.html" rel="noreferrer">sched</a> module:</p>
  
  <pre><code># sc is a sched.scheduler instance
  sc.enter(60, 1, self.doChecks, (sc, False))
  </code></pre>
  
  <p>The script is running as a daemonised process using the code <a href="http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/" rel="noreferrer">here</a>.</p>
  
  <p>A number of class methods that are called as part of doChecks use the <a href="http://docs.python.org/library/subprocess.html" rel="noreferrer">subprocess</a> module to call system functions in order to get system statistics:</p>
  
  <pre><code>ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]
  </code></pre>
  
  <p>This runs fine for a period of time before the entire script crashing with the following error:</p>
  
  <pre><code>File "/home/admin/sd-agent/checks.py", line 436, in getProcesses
  File "/usr/lib/python2.4/subprocess.py", line 533, in __init__
  File "/usr/lib/python2.4/subprocess.py", line 835, in _get_handles
  OSError: [Errno 12] Cannot allocate memory
  </code></pre>
  
  <p>The output of free -m on the server once the script has crashed is:</p>
  
  <pre><code>$ free -m
                    total       used       free     shared     buffers    cached
  Mem:                894        345        549          0          0          0
  -/+ buffers/cache:  345        549
  Swap:                 0          0          0
  </code></pre>
  
  <p>The server is running CentOS 5.3. I am unable to reproduce on my own CentOS boxes nor with any other user reporting the same problem.</p>
  
  <p>I have tried a number of things to debug this as suggested in the original question:</p>
  
  <ol>
  <li><p>Logging the output of free -m before and after the Popen call. There is no significant change in memory usage i.e. memory is not gradually being used up as the script runs.</p></li>
  <li><p>I added close_fds=True to the Popen call but this made no difference - the script still crashed with the same error. Suggested <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1297785#1297785">here</a> and <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1278961#1278961">here</a>.</p></li>
  <li><p>I checked the rlimits which showed (-1, -1) on both RLIMIT_DATA and RLIMIT_AS as suggested <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1270442#1270442">here</a>.</p></li>
  <li><p><a href="http://www.zenoss.com/community/wiki/common-error-messages-and-solutions/oserror-errno-12-cannot-allocate-memory-in-popen2.py/" rel="noreferrer">An article</a> suggested the having no swap space might be the cause but swap is actually available on demand (according to the web host) and this was also suggested as a bogus cause <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1270171#1270171">here</a>.</p></li>
  <li><p>The processes are being closed because that is the behaviour of using .communicate() as backed up by the Python source code and comments <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1216824#1216824">here</a>.</p></li>
  </ol>
  
  <p>The entire checks can be found at on <a href="http://github.com/dmytton/sd-agent/blob/82f5ff9203e54d2adeee8cfed704d09e3f00e8eb/checks.py" rel="noreferrer">GitHub here</a> with the getProcesses function defined from line 442. This is called by doChecks() starting at line 520.</p>
  
  <p>The script was run with strace with the following output before the crash:</p>
  
  <pre><code>recv(4, "Total Accesses: 516662\nTotal kBy"..., 234, 0) = 234
  gettimeofday({1250893252, 887805}, NULL) = 0
  write(3, "2009-08-21 17:20:52,887 - checks"..., 91) = 91
  gettimeofday({1250893252, 888362}, NULL) = 0
  write(3, "2009-08-21 17:20:52,888 - checks"..., 74) = 74
  gettimeofday({1250893252, 888897}, NULL) = 0
  write(3, "2009-08-21 17:20:52,888 - checks"..., 67) = 67
  gettimeofday({1250893252, 889184}, NULL) = 0
  write(3, "2009-08-21 17:20:52,889 - checks"..., 81) = 81
  close(4)                                = 0
  gettimeofday({1250893252, 889591}, NULL) = 0
  write(3, "2009-08-21 17:20:52,889 - checks"..., 63) = 63
  pipe([4, 5])                            = 0
  pipe([6, 7])                            = 0
  fcntl64(7, F_GETFD)                     = 0
  fcntl64(7, F_SETFD, FD_CLOEXEC)         = 0
  clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)
  write(2, "Traceback (most recent call last"..., 35) = 35
  open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  write(2, "  File \"/usr/bin/sd-agent/agent."..., 52) = 52
  open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  write(2, "  File \"/home/admin/sd-agent/dae"..., 60) = 60
  open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  write(2, "  File \"/usr/bin/sd-agent/agent."..., 54) = 54
  open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8
  write(2, "  File \"/usr/lib/python2.4/sched"..., 55) = 55
  fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0
  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
  read(8, "\"\"\"A generally useful event sche"..., 4096) = 4054
  write(2, "    ", 4)                     = 4
  write(2, "void = action(*argument)\n", 25) = 25
  close(8)                                = 0
  munmap(0xb7d28000, 4096)                = 0
  open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  write(2, "  File \"/usr/bin/sd-agent/checks"..., 60) = 60
  open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
  open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
  write(2, "  File \"/usr/bin/sd-agent/checks"..., 64) = 64
  open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
  write(2, "  File \"/usr/lib/python2.4/subpr"..., 65) = 65
  fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
  read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
  read(8, "lso, the newlines attribute of t"..., 4096) = 4096
  read(8, "code &lt; 0:\n        print &gt;&gt;sys.st"..., 4096) = 4096
  read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
  read(8, " p2cread\n        # c2pread    &lt;-"..., 4096) = 4096
  write(2, "    ", 4)                     = 4
  write(2, "errread, errwrite)\n", 19)    = 19
  close(8)                                = 0
  munmap(0xb7d28000, 4096)                = 0
  open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
  write(2, "  File \"/usr/lib/python2.4/subpr"..., 71) = 71
  fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
  mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
  read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
  read(8, "lso, the newlines attribute of t"..., 4096) = 4096
  read(8, "code &lt; 0:\n        print &gt;&gt;sys.st"..., 4096) = 4096
  read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
  read(8, " p2cread\n        # c2pread    &lt;-"..., 4096) = 4096
  read(8, "table(self, handle):\n           "..., 4096) = 4096
  read(8, "rrno using _sys_errlist (or siml"..., 4096) = 4096
  read(8, " p2cwrite = None, None\n         "..., 4096) = 4096
  write(2, "    ", 4)                     = 4
  write(2, "self.pid = os.fork()\n", 21)  = 21
  close(8)                                = 0
  munmap(0xb7d28000, 4096)                = 0
  write(2, "OSError", 7)                  = 7
  write(2, ": ", 2)                       = 2
  write(2, "[Errno 12] Cannot allocate memor"..., 33) = 33
  write(2, "\n", 1)                       = 1
  unlink("/var/run/sd-agent.pid")         = 0
  close(3)                                = 0
  munmap(0xb7e0d000, 4096)                = 0
  rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0
  brk(0xa022000)                          = 0xa022000
  exit_group(1)                           = ?
  </code></pre>