title: How to log to single file with multiprocessing.Pool.apply_async
body: <p>I can't get logging to a single file working with multprocess.Pool.apply_async.
  I'm trying to adapt <a href="https://docs.python.org/3.5/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes" rel="nofollow noreferrer">this</a> example from the Logging Cookbook, but it only works for <code>multiprocessing.Process</code>. Passing the logging queue into <code>apply_async</code> doesn't seem to have effect.
  I would like to use a Pool so that I can easily manage the number of simultaneous threads.</p>
  
  <p>The following adapted example with multiprocessing.Process works ok for me, except I am not getting log messages from the main process, and I don't think it will work well when I have 100 large jobs.</p>
  
  <pre><code>import logging
  import logging.handlers
  import numpy as np
  import time
  import multiprocessing
  import pandas as pd
  log_file = 'PATH_TO_FILE/log_file.log'
  
  def listener_configurer():
      root = logging.getLogger()
      h = logging.FileHandler(log_file)
      f = logging.Formatter('%(asctime)s %(processName)-10s %(name)s %(levelname)-8s %(message)s')
      h.setFormatter(f)
      root.addHandler(h)
  
  # This is the listener process top-level loop: wait for logging events
  # (LogRecords)on the queue and handle them, quit when you get a None for a
  # LogRecord.
  def listener_process(queue, configurer):
      configurer()
      while True:
          try:
              record = queue.get()
              if record is None:  # We send this as a sentinel to tell the listener to quit.
                  break
              logger = logging.getLogger(record.name)
              logger.handle(record)  # No level or filter logic applied - just do it!
          except Exception:
              import sys, traceback
              print('Whoops! Problem:', file=sys.stderr)
              traceback.print_exc(file=sys.stderr)
  
  
  def worker_configurer(queue):
      h = logging.handlers.QueueHandler(queue)  # Just the one handler needed
      root = logging.getLogger()
      root.addHandler(h)
      # send all messages, for demo; no other level or filter logic applied.
      root.setLevel(logging.DEBUG)
  
  
  # This is the worker process top-level loop, which just logs ten events with
  # random intervening delays before terminating.
  # The print messages are just so you know it's doing something!
  def worker_function(sleep_time, name, queue, configurer):
      configurer(queue)
      start_message = 'Worker {} started and will now sleep for {}s'.format(name, sleep_time)
      logging.info(start_message)
      time.sleep(sleep_time)
      success_message = 'Worker {} has finished sleeping for {}s'.format(name, sleep_time)
      logging.info(success_message)
  
  def main_with_process():
      start_time = time.time()
      single_thread_time = 0.
      queue = multiprocessing.Queue(-1)
      listener = multiprocessing.Process(target=listener_process,
                                         args=(queue, listener_configurer))
      listener.start()
      workers = []
      for i in range(10):
          name = str(i)
          sleep_time = np.random.randint(10) / 2
          single_thread_time += sleep_time
          worker = multiprocessing.Process(target=worker_function,
                                           args=(sleep_time, name, queue, worker_configurer))
          workers.append(worker)
          worker.start()
      for w in workers:
          w.join()
      queue.put_nowait(None)
      listener.join()
      end_time = time.time()
      final_message = "Script execution time was {}s, but single-thread time was {}s".format(
          (end_time - start_time),
          single_thread_time
      )
      print(final_message)
  
  if __name__ == "__main__":
      main_with_process()
  </code></pre>
  
  <p>But I can't get the following adaptation to work: </p>
  
  <pre><code>def main_with_pool():
      start_time = time.time()
      queue = multiprocessing.Queue(-1)
      listener = multiprocessing.Process(target=listener_process,
                                         args=(queue, listener_configurer))
      listener.start()
      pool = multiprocessing.Pool(processes=3)
      job_list = [np.random.randint(10) / 2 for i in range(10)]
      single_thread_time = np.sum(job_list)
      for i, sleep_time in enumerate(job_list):
          name = str(i)
          pool.apply_async(worker_function,
                           args=(sleep_time, name, queue, worker_configurer))
  
      queue.put_nowait(None)
      listener.join()
      end_time = time.time()
      print("Script execution time was {}s, but single-thread time was {}s".format(
          (end_time - start_time),
          single_thread_time
      ))
  
  if __name__ == "__main__":
      main_with_pool()
  </code></pre>
  
  <p>I've tried many slight variations, using multiprocessing.Manager, multiprocessing.Queue, multiprocessing.get_logger, apply_async.get(), but haven't gotten any to work.</p>
  
  <p>I would think there would be an off-the-shelf solution for this. Should I try Celery instead?</p>
  
  <p>thanks</p>