A simple, multiprocess-safe logger for Python
Python's built-in loggers are pretty handy - they're easily customized and come with useful functionality out of the box, including things like file rotation. These file handlers are thread-safe, but not process-safe, so, if you're attempting to write to a common log file from multiple processes (e.g. in a pre-forking web server) you run the risk of your workers trampling over each other. File locking is a possible workaround, but that's yucky.
To avoid this, it is recommended that one uses a socket-based logger (a code sample is helpfully provided in the Logging Cookbook). However, it is just a code snippet. Multilog is a dependency-free implementation of the sample socket logger with some niceties, like fileConfig support, and parameterization.
Once installed, the Multilog daemon can be invoked via:
mutlilogUsage:
usage: multilog [-h] [-s SERVER] [-p PORT] [-c CONFIG_PATH]
A simple logger for multiple Python processes.
optional arguments:
-h, --help show this help message and exit
-s SERVER, --server SERVER
The server hostname (default: localhost)
-p PORT, --port PORT The port to listen on. (default: 9020)
-c CONFIG_PATH, --config CONFIG_PATH
The log configuration to load. (default: logging.ini)By default, it will look for a logging.ini file in the current directory. If one isn't found, Multilog will
yell at you. A sample configuration file for the server:
[loggers]
keys=root
[handlers]
keys=multilogServerHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=NOTSET
handlers=multilogServerHandler
[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')
[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]and for your application:
[loggers]
keys=root
[handlers]
keys=multilogClientHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=NOTSET
handlers=multilogClientHandler
[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)
[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]The important field is the args block in the handler_multilogClientHandler section - those parameters should correspond to the server and ports on which the multilog daemon is listening. By default, the daemon uses localhost and logging.handlers.DEFAULT_TCP_LOGGING_PORT.
If you want to have Multilog share your application's config, you can do the following:
[loggers]
keys=root,appName
[handlers]
keys=multilogClientHandler,multilogServerHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=NOTSET
handlers=%(root_handler)s
[logger_appName]
level=INFO
handlers=
propagate=1
qualname=appName
[handler_multilogClientHandler]
class=handlers.SocketHandler
level=DEBUG
formatter=simpleFormatter
args=('localhost', handlers.DEFAULT_TCP_LOGGING_PORT)
[handler_multilogServerHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/appName/appName.log', 'midnight')
[formatter_simpleFormatter]
class=logging.Formatter
format=%(asctime)s %(levelname)7s: PID: %(process)5s | %(message)s [in %(pathname)s:%(lineno)d]Then, in your application, pass the root handler name into the logging config:
import logging
logging.config.fileConfig(config_path, defaults={"root_handler": "multilogClientHandler"})Multilog will always load the multilogServerHandler handler. If you don't want to run Multilog (if you're running a single-threaded local dev server, for example), simply change your root_handler value to multilogServerHandler to write to the handler.
Multilog is compatible with Python 2.6, 2.7, and 3.3+.