title: python web.py execute external program asynchronously
body: <p>I have a web.py application running on built in cherrypy server.
  I want to execute an external script when user post to an url, and the script will run in the background in a python subprocess.Popen call and web.py will redirect to another page where the progress of the script is being monitored using a jquery ajax request.
  But I'm unable to properly implement this here.
  the code snippets are as follows, I can provide more information if needed.</p>
  
  <pre><code>        import web
      from multiprocessing import Process
      import subprocess, shlex
      import time, json
      from login import authorize, sessidGen
  
      def __callProcess(processString,mod='w',shell=False):
          if not shell: args = shlex.split(processString)
          else: args = processString
          out = open('./bteq/logs/output.log',mod)
          err = open('./bteq/logs/error.log',mod)
          p = subprocess.Popen(args,stdout=out,stderr=err,shell=shell)
          return p
  
      def setExec():
          __callProcess("chmod +x ./bteq/*.sh",shell=True)
  
      def bteqExec(filename,system):
          if system not in ['prod','da','cdw','cdw2','cert','']: return False
          processString = " ".join([filename,system])
          p = __callProcess(processString)
          return p.pid
  
      render = web.template.render('templates/',base='layout')
      render_plain = web.template.render('templates/')
  
      class Executor:     
          def GET(self):
              authorize()
              session = web.ctx.session
              inputs = web.input(sessid={},type={})
              if not inputs.sessid or session.id != inputs.sessid: web.seeother('/')
              if inputs.sessid and inputs.type:
                  return render.executor('BTEQ Executor',inputs.type,inputs.sessid)
              else: raise web.seeother('/')
  
          def POST(self):
              authorize()
              session = web.ctx.session
              inputs = web.input(sessid={},type={},act={})
              if not inputs.sessid or session.id != inputs.sessid: web.seeother('/')
              if inputs and inputs.act == 'start':
                  pid = bteqExec('python ./bteq/timer.py','')
                  session.id = sessidGen()
                  session.exctrpid = pid
                  return web.seeother('/progress.htm')
              else: raise web.seeother('/')
  
      class progress:
          def GET(self):
              authorize()
              session = web.ctx.session
              inputs = web.input(ajax={})
              if inputs.ajax == 'true': 
                  web.header('Content-Type', 'application/json')
                  if session.count &gt;= 100: session.count = 0
                  session.count += 10
                  pid = session.exctrpid
                  out = open('./bteq/logs/output.log','r')
                  err = open('./bteq/logs/error.log','r')
                  output = ('&lt;strong&gt;OUTPUT:&lt;/strong&gt;&lt;br&gt;'+out.read()).replace('\n','&lt;br&gt;')
                  err = err.read()
                  if err:error = ('&lt;strong&gt;ERRORS:&lt;/strong&gt;&lt;br&gt;'+err.read()).replace('\n','&lt;br&gt;')
                  else: error = None
                  d = {'count':session.count,'msg':output,'err':error,'rc':pid,'session_id':session.session_id}
                  return json.dumps(d)
              r = web.template.Template('$def with (title)\n$var title: $title\n')
              return render_plain.layout_pgbar(r('progress test'))
  </code></pre>
  
  <p>Since subprocess.Popen object is not pickleable it is not able to be put in as a session variable, and I want to the p.poll() and p.stdout.read() from the progress class.</p>
  
  <p>Also I want the code runnable in both linux and windows, I have my dev setup in windows and I deploy it on linux servers.</p>
  
  <p>Could someone help me out here...</p>
  
  <p>Thank you.</p>