title: logging while using parallel python
body: <p>I'm using parallel python to execute a big function (<code>executePipeline</code>) multiple times. This function is also using multiprocessing (with the <code>multiprocessing</code> module).<br>
  I'm having some trouble to display correctly the logging messages on my console using the parallel python module. When I'm not using it, the logging messages are well displayed.</p>
  
  <p>Here is how it works. I have a server who is calling a worker every time it gets a request from  client using: </p>
  
  <pre><code>job = self.server.job_server.submit(func = executeWorker, args = (config, ) )
  </code></pre>
  
  <p>This function is execute from a new thread every time there is a new request from a client.
  Then the worker is calling the function <code>executePipeline</code> which is executing different processus using multiprocessing.</p>
  
  <p>The server a <code>SocketServer.TCPServer</code> I using threading. I set up a logger in my server as following using the root logger:</p>
  
  <pre><code>self.logger = logging.getLogger()
  self.logger.setLevel(logging.INFO)
  self.logger.addHandler(logging.StreamHandler() 
  self.job_server = pp.Server(ncpus = 8) # for test
  self.jobs = []
  </code></pre>
  
  <p>When I'm running my server i can only get the logging from the <code>executePipeline</code> but not from the child processes.
  Also I get the logging of the execute pipeline only at the end of the job not while is running.</p>
  
  <p>Also here is the worker code. The <code>"Executing pipeline with worker number "</code> is well displayed in my terminal</p>
  
  <pre><code>'''
  Setup logging
  '''
  
  logger = logging.getLogger()
  logger.setLevel(logging.INFO)  
  
  # worker name
  publicIP = socket.gethostbyname(socket.gethostname()) 
  pid = os.getpid()
  workerID = unicode(str(publicIP) + ":" + str(pid))
  
  logger.info( "Executing pipeline with worker {}".format(workerID))
  res = executePipeline(config)    
  markedScore = res["marker.score"]
  markedDetails = res["marker.detail"]
  results = {'marker.detail' : markedDetails , 'marker.score' : markedScore } 
  
  return results
  </code></pre>
  
  <p>Is there a good way too get the logging working properly and see what the child process of my <code>executePipeline</code> function are sending back?</p>
  
  <p>Thanks for your help!</p>
  
  <p>Romanzo</p>