title: Python logging to multiple handlers, at different log levels?
body: <p>Folks,</p>
  
  <p>I'm scratching my head on a python logging config that I can't get right.</p>
  
  <p>Let's say I have the following package installed:</p>
  
  <pre><code>mypackage/
     data/mypackage.logging.conf
     module1.py
     module2.py
     scripts/main.py
  </code></pre>
  
  <p>As the script can be used interactively or run from a crontab, I have the following requirements:</p>
  
  <ol>
  <li><p>no print statements, everything goes through logging;</p></li>
  <li><p>log using a <code>timedRotatingFileHandler</code>, <em>always</em> at DEBUG level;</p></li>
  <li><p>log using a <code>mailinglogger.SummarisingLogger</code>, <em>always</em> at INFO level;</p></li>
  <li><p>log to console, with level set by default to INFO or overridden through a command-line option.</p></li>
  </ol>
  
  <p>Problem is, I can change the log level through the command-line and the console log level is changed accordingly, but the other handlers are also changed, which I don't want... :-/</p>
  
  <p>In a logging config file, I'm not sure I understand the precedence between the root logger's level, other loggers' level and handlers' level settings.</p>
  
  <p>Here is some sample code. Any clues will be appreciated :-)</p>
  
  <pre><code># mypackage/data/mypackage.logging.conf
  [loggers]
  root,mypackage
  
  [handlers]
  keys=consoleHandler,timedRotatingFileHandler,summarisingHandler
  
  [formatters]
  keys=simpleFormatter,consoleFormatter,mypackageFormatter
  
  [logger_root]
  #level=INFO
  handlers=consoleHandler
  
  [logger_mypackage]
  #level=INFO
  handlers=timedRotatingFileHandler,summarisingHandler
  qualname=mypackage
  
  [handler_consoleHandler]
  class=StreamHandler
  #level=INFO
  formatter=consoleFormatter
  args=(sys.stdout,)
  
  [handler_timedRotatingFileHandler]
  class=logging.handlers.TimedRotatingFileHandler
  level=DEBUG
  formatter=mypackageFormatter
  args=('mypackage.log', 'M', 1, 5)
  
  [handler_summarisingHandler]
  class=mailinglogger.SummarisingLogger
  level=INFO
  formatter=mypackageFormatter
  args=('mypackage@someserver.somewhere.com', ('mypackage-alerts@somewhere.com',), 'relay.somewhere.com')
  </code></pre>
  
  <hr>
  
  <pre><code>#mypackage/scripts/main.py:
  import logging
  import logging.config
  import os
  import sys
  
  import mypackage.module1
  import mypackage.module2
  
  logging.config.fileConfig('data/mypackage.logging.conf')
  log = logging.getLogger(__name__)
  
  if __name__ == '__main__':
      loglevel = 'INFO'
      if len(sys.argv) &gt; 1:
          loglevel = sys.argv[1].upper()
  
      logging.getLogger('').setLevel(getattr(logging, loglevel))
      # or logging.getLogger('mypackage').setLevel(getattr(logging, loglevel)) ?
  
      mypackage.module1.do_something()
      mypackage.module2.do_something_else()
  </code></pre>
  
  <hr>
  
  <pre><code>#mypackage/module1.py:
  import logging
  
  log = logging.getLogger(__name__)
  log.addHandler(NullHandler())
  
  def do_something():
      log.debug("some debug message from:" + __name__)
      log.info("some info message from:" + __name__)
      log.error("some error message from:" + __name__)
  </code></pre>
  
  <hr>
  
  <pre><code>#mypackage/module2.py:
  import logging
  
  log = logging.getLogger(__name__)
  log.addHandler(NullHandler())
  
  def do_something_else():
      log.debug("some debug message from:" + __name__)
      log.info("some info message from:" + __name__)
      log.error("some error message from:" + __name__)
  </code></pre>
  
  <hr>
  
  <p><strong>UPDATE 1</strong></p>
  
  <p>In the meantime, I discovered <a href="https://stackoverflow.com/a/11111212/131036">this answer</a> and successfully modified my code this way:</p>
  
  <pre><code>#mypackage/scripts/main.py:
  import logging
  import logging.config
  import os
  import sys
  import mailinglogger
  
  import mypackage.module1
  import mypackage.module2
  
  def main():
      # get the console log level from the command-line
      loglevel = 'INFO'
      if len(sys.argv) &gt; 1:
          loglevel = sys.argv[1].upper()
  
      # create formatters
      simple_formatter = logging.Formatter("%(name)s:%(levelname)s: %(message)s")
      detailed_formatter = logging.Formatter("%(asctime)s %(name)s[%(process)d]: %(levelname)s - %(message)s")
  
      # get a top-level "mypackage" logger,
      # set its log level to DEBUG,
      # BUT PREVENT IT from propagating messages to the root logger
      #
      log = logging.getLogger('mypackage')
      log.setLevel(logging.DEBUG)
      log.propagate = 0
  
      # create a console handler
      # and set its log level to the command-line option 
      # 
      console_handler = logging.StreamHandler(sys.stdout)
      console_handler.setLevel(getattr(logging, loglevel))
      console_handler.setFormatter(simple_formatter)
  
      # create a file handler
      # and set its log level to DEBUG
      #
      file_handler = logging.handlers.TimedRotatingFileHandler('mypackage.log', 'M', 1, 5)
      file_handler.setLevel(logging.DEBUG)
      file_handler.setFormatter(detailed_formatter)
  
      # create a mail handler
      # and set its log level to INFO
      #
      mail_handler = mailinglogger.SummarisingLogger(
          'mypackage@someserver.somewhere.com', ('mypackage-alerts@somewhere.com',), 'relay.somewhere.com')
      mail_handler.setLevel(logging.INFO)
      mail_handler.setFormatter(detailed_formatter)
  
      # add handlers to the "mypackage" logger
      #
      log.addHandler(console_handler)
      log.addHandler(file_handler)
      log.addHandler(mail_handler)
  
      # let the modules do their stuff 
      # and log to the "mypackage.module1" and "mypackage.module2" loggers
      #
      mypackage.module1.do_something()
      mypackage.module2.do_something_else()
  
  
  if __name__ == '__main__':
      main()
  </code></pre>
  
  <p>Now, I'll try to translate this in a logging.config file...</p>
  
  <hr>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p>Here is the best logging config and code combination I found.</p>
  
  <p>In the mypackage.logging.conf file, the "mypackage" logger is:</p>
  
  <ul>
  <li>set up only with the file and email handlers;</li>
  <li>its propagate is set to false;</li>
  <li>its level is set to DEBUG;</li>
  <li>while the file and email handlers are respectively set to INFO and DEBUG.</li>
  </ul>
  
  <hr>
  
  <pre><code>#mypackage/data/mypackage.logging.conf
  [loggers]
  keys=root,mypackage
  
  [handlers]
  keys=consoleHandler,timedRotatingFileHandler,summarisingHandler
  
  [formatters]
  keys=simpleFormatter,consoleFormatter,mypackageFormatter
  
  [logger_root]
  #level=INFO
  handlers=consoleHandler
  
  [logger_mypackage]
  level=DEBUG
  handlers=timedRotatingFileHandler,summarisingHandler
  qualname=mypackage
  propagate=0
  
  [handler_consoleHandler]
  class=StreamHandler
  #level=INFO
  formatter=consoleFormatter
  args=(sys.stdout,)
  
  [handler_timedRotatingFileHandler]
  class=logging.handlers.TimedRotatingFileHandler
  level=DEBUG
  formatter=mypackageFormatter
  args=('mypackage.log', 'M', 1, 5)
  
  [handler_summarisingHandler]
  class=mailinglogger.SummarisingLogger
  level=INFO
  formatter=mypackageFormatter
  args=('mypackage@someserver.somewhere.com', ('mypackage-alerts@somewhere.com',), 'relay.somewhere.com')
  
  [formatter_consoleFormatter]
  format=%(levelname)s: %(message)s
  datefmt=
  
  [formatter_mypackageFormatter]
  format=%(asctime)s %(name)s[%(process)d]: %(levelname)s - %(message)s
  datefmt=
  </code></pre>
  
  <hr>
  
  <p>In the script:</p>
  
  <ol>
  <li><p>the logging config is read;</p></li>
  <li><p>a console_formatter is (re-)created;</p></li>
  <li><p>a console handler is created with the log level from the command-line option, then added to the "mypackage" logger.</p></li>
  </ol>
  
  <hr>
  
  <pre><code>import logging
  import logging.config
  import os
  import sys
  
  import mypackage.module1
  import mypackage.module2
  
  def setup_logging(loglevel):
      #
      # load logging config from file
      #
      logging.config.fileConfig('data/mypackage.logging.conf')
  
      # (re-)create formatter
      console_formatter = logging.Formatter("%(name)s:%(levelname)s: %(message)s")
  
      # create a console handler
      # and set its log level to the command-line option 
      # 
      console_handler = logging.StreamHandler(sys.stdout)
      console_handler.setFormatter(console_formatter)
      console_handler.setLevel(getattr(logging, loglevel))
  
      # add console handler to the pre-configured "mypackage" logger
      #
      logger = logging.getLogger('mypackage')
      logger.addHandler(console_handler)
  
  
  def main():
      # get the console log level from the command-line
      loglevel = 'INFO'
      if len(sys.argv) &gt; 1:
          loglevel = sys.argv[1].upper()
  
      # load logging config and setup console handler
      #
      setup_logging(loglevel)
  
      # log from the script to the "mypackage.scripts.main" logger
      #
      log = logging.getLogger(__name__)
      log.debug("some debug message from:" + __name__)
      log.info("some info message from:" + __name__)
      log.error("some error message from:" + __name__)
  
      # let the modules do their stuff 
      # and log to the "mypackage.module1" and "mypackage.module2" loggers
      #
      mypackage.module1.do_something()
      mypackage.module2.do_something_else()
  
  if __name__== '__main__':
      main()
  </code></pre>
  
  <hr>
  
  <p>Things would be simpler if the handlers were "addressable" by name when loaded from a config file.</p>
  
  <p>Then, we could have the mypackage console handler set up in the config file and its log level changed in the code like this:</p>
  
  <pre><code>def setup_logging(loglevel):
      logging.config.fileConfig('data/mypackage.logging.conf')
  
      logger = logging.getLogger('mypackage')
      console_handler = logger.getHandler('consoleHandler')
      console_handler.setLevel(getattr(logging, loglevel))
  </code></pre>
  
  <p>There would no need to re-create a formatter either...</p>
  
  <p>(last update: yes, I'm aware of <a href="https://docs.python.org/3/library/logging.config.html#incremental-configuration" rel="noreferrer">https://docs.python.org/3/library/logging.config.html#incremental-configuration</a>, but in this case, I'm stuck with Python 2.6... :-)</p>