Integrating python-json-logger with Python’s logging framework

Integrating python-json-logger with Python’s logging framework

One of the key challenges in logging and log analysis is making log data readable by machines. The default output format of the Python logging framework is not machine-friendly. To address this, the python-json-logger library provides a solution by allowing standard Python logging to output log data as JSON objects. In this article, we will explore how to integrate python-json-logger with Python’s logging framework and customize the log output.

Installing python-json-logger

You can install python-json-logger using pip:

#bash
pip install python-json-logger

Integrating with Python’s logging framework

To integrate python-json-logger with the Python logging framework, you need to add a custom formatter to your logging configuration. Here’s an example code snippet:

#python
import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger()

logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)

This code snippet configures a JSON formatter and attaches it to the logger’s stream handler. Now, when you log messages using the Python logging framework, the output will be in JSON format.

Customizing fields

The python-json-logger library allows you to customize the JSON fields in the log output. You can override the default formatter’s parse() method to specify required fields. Here’s an example that demonstrates overriding the formatter:

#python
class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def parse(self):
        return self._fmt.split(';')

formatter = CustomJsonFormatter('one;two')

In this example, we create a custom formatter that adds two fields, one and two, to the log output. You can also use the %() syntax to specify fields directly in the log format string.

Adding custom object serialization

Sometimes, you may need to handle custom object serialization in the log output. python-json-logger allows you to specify a default JSON object translator or provide a custom encoder. Here’s an example:

#python
def json_translate(obj):
    if isinstance(obj, MyClass):
        return {"special": obj.special}

formatter = jsonlogger.JsonFormatter(json_default=json_translate, json_encoder=json.JSONEncoder)
logHandler.setFormatter(formatter)

logger.info({"special": "value", "run": 12})
logger.info("classic message", extra={"special": "value", "run": 12})

In this example, we define a json_translate function that handles serialization for a custom class MyClass. We then pass this function to the JSON formatter along with a custom JSON encoder.

Using a Config File

If you prefer to use a configuration file to configure the python-json-logger, you can do so by using the fileConfig function. Here’s an example of a config file:

#ini
[loggers]
keys = root,custom

[logger_root]
handlers =

[logger_custom]
level = INFO
handlers = custom
qualname = custom

[handlers]
keys = custom

[handler_custom]
class = StreamHandler
level = INFO
formatter = json
args = (sys.stdout,)

[formatters]
keys = json

[formatter_json]
format = %(message)s
class = pythonjsonlogger.jsonlogger.JsonFormatter

This configuration file sets up a StreamHandler with the JSON formatter, and the log output is directed to sys.stdout.

Conclusion

Integrating python-json-logger with Python’s logging framework allows you to output log data as JSON objects, making it more readable by machines. By customizing the log fields and handling custom object serialization, you can further tailor the log output to your needs. Start using python-json-logger today and take your logging to the next level.

External Examples

• Wesley Tanaka – Structured log files in Python using python-json-logger

• Archive

In this article, we covered how to integrate python-json-logger with Python’s logging framework, customize the log output, and handle custom object serialization. The integration of python-json-logger with Python’s logging framework is beneficial for several reasons:

1. Readability by Machines: By outputting log data as JSON objects, it becomes easier for machines to parse and process the logs. This simplifies log analysis and enables better log monitoring and alerting.

2. Standardized Format: python-json-logger provides a standardized format for log data. This eliminates the need for writing custom parsers for syslog type records, saving development time and effort.

3. Customization: python-json-logger allows you to customize the fields in the log output. You can add extra fields, specify required fields, and even handle custom object serialization. This flexibility enables you to tailor the log output to your specific requirements.

By integrating python-json-logger with other software systems such as Docker, databases like MongoDB and MySQL, and web frameworks like FastAPI and Django, you can enhance the capabilities of your logging infrastructure and enable seamless integration with your existing technology stack.