LoggerJSON

NOTE: This is a fork of the excellent Nebo15/logger_json logger with some additional PRs merged in to support Stord's usage with DataDog. The eventual goal is to replace this with the official OpenTelemetry logger implementation for Elixir.

JSON console back-end for Elixir Logger.

It can be used as drop-in replacement for default :console Logger back-end in cases where you use Google Cloud Logger, DataDog or other JSON-based log collectors. After adding this back-end you may also be interested in redirecting otp and sasl reports to Logger (see "Error Logger configuration" section).

Minimum supported Erlang/OTP version is 20.

Log Format

LoggerJSON provides three JSON formatters out of the box and allows developers to implement a custom one.

BasicLogger

The LoggerJSON.Formatters.BasicLogger formatter provides a generic JSON formatted message with no vendor specific entries in the payload. A sample log entry from LoggerJSON.Formatters.BasicLogger looks like the following:

{
  "time": "2020-04-02T11:59:06.710Z",
  "severity": "debug",
  "message": "hello",
  "metadata": {
    "user_id": 13
  }
}

GoogleCloudLogger

Generates JSON that is compatible with the Google Cloud Logger LogEntry format:

{
  "message":"hello",
  "logging.googleapis.com/sourceLocation":{
    "file":"/os/logger_json/test/unit/logger_json_test.exs",
    "function":"Elixir.LoggerJSONGoogleTest.test metadata can be configured/1",
    "line":71
  },
  "severity":"DEBUG",
  "time":"2018-10-19T01:10:49.582Z",
  "user_id":13
}

Notice that GKE doesn't allow to set certain fields of the LogEntry, so support is limited. The results in Google Cloud Logger would looks something like this:

{
  "httpRequest":{
    "latency":"0.350s",
    "remoteIp":"::ffff:10.142.0.2",
    "requestMethod":"GET",
    "requestPath":"/",
    "requestUrl":"http://10.16.0.70/",
    "status":200,
    "userAgent":"kube-probe/1.10+"
  },
  "insertId":"1g64u74fgmqqft",
  "jsonPayload":{
    "message":"",
    "phoenix":{
      "action":"index",
      "controller":"Elixir.MyApp.Web.PageController",
    },
    "request_id":"2lfbl1r3m81c40e5v40004c2",
    "vm":{
      "hostname":"myapp-web-66979fc-vbk4q",
      "pid":1,
    }
  },
  "logName":"projects/hammer-staging/logs/stdout",
  "metadata":{
    "systemLabels":{},
    "userLabels":{}
  },
  "operation":{
    "id":"2lfbl1r3m81c40e5v40004c2"
  },
  "receiveTimestamp":"2018-10-18T14:33:35.515253723Z",
  "resource":{},
  "severity":"INFO",
  "sourceLocation":{
    "file":"iex",
    "function":"Elixir.LoggerJSON.Plug.call/2",
    "line":"36"
  },
  "timestamp":"2018-10-18T14:33:33.263Z"
}

DatadogLogger

Adheres to the default standard attribute list.

{
  "domain": ["elixir"],
  "duration": 3863403,
  "http": {
    "url": "http://localhost/create-account",
    "status_code": 200,
    "method": "GET",
    "referer": "http://localhost:4000/login",
    "request_id": "http_FlDCOItxeudZJ20AAADD",
    "useragent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36",
    "url_details": {
      "host": "localhost",
      "port": 4000,
      "path": "/create-account",
      "queryString": "",
      "scheme": "http"
    }
  },
  "logger": {
    "thread_name": "#PID<0.1042.0>",
    "method_name": "Elixir.LoggerJSON.Plug.call/2"
  },
  "message": "",
  "network": {
    "client": {
      "ip": "127.0.0.1"
    }
  },
  "phoenix": {
    "controller": "Elixir.RecognizerWeb.Accounts.UserRegistrationController",
    "action": "new"
  },
  "request_id": "http_FlDCOItxeudZJ20AAADD",
  "syslog": {
    "hostname": [10, 10, 100, 100, 100, 100, 100],
    "severity": "info",
    "timestamp": "2020-12-14T19:16:55.088Z"
  }
}

Custom formatters

You can change this structure by implementing LoggerJSON.Formatter behaviour and passing module name to :formatter config option. Example module can be found in LoggerJSON.Formatters.GoogleCloudLogger.

config :logger_json, :backend,
  formatter: MyFormatterImplementation

Installation

It's available on Hex, the package can be installed as:

1. Add :logger_json to your list of dependencies in mix.exs:

def deps do
  [{:logger_json, "~> 5.1"}]
end

2. Set configuration in your config/config.exs:

config :logger_json, :backend,
  metadata: :all,
  json_encoder: Jason,
  formatter: LoggerJSON.Formatters.GoogleCloudLogger

Some integrations (for eg. Plug) use metadata to log request and response parameters. You can reduce log size by replacing :all (which means log all metadata) with a list of the ones that you actually need.

Beware that LoggerJSON always ignores some metadata keys, but formatters like GoogleCloudLogger and DatadogLogger still persist those metadata values into a structured output. This behavior is similar to the default Elixir logger backend.

3. Replace default Logger :console back-end with LoggerJSON:

config :logger,
  backends: [LoggerJSON]

4. Optionally. Log requests and responses by replacing a Plug.Logger in your endpoint with a:

plug LoggerJSON.Plug

LoggerJSON.Plug is configured by default to use LoggerJSON.Plug.MetadataFormatters.GoogleCloudLogger. You can replace it with the :metadata_formatter config option.

5. Optionally. Use Ecto telemetry for additional metadata:

Attach telemetry handler for Ecto events in start/2 function in application.ex

:ok =
  :telemetry.attach(
    "logger-json-ecto",
    [:my_app, :repo, :query],
    &LoggerJSON.Ecto.telemetry_logging_handler/4,
    :debug
  )

Prevent duplicate logging of events, by setting log configuration option to false

config :my_app, MyApp.Repo,
  adapter: Ecto.Adapters.Postgres,
  log: false

Dynamic configuration

For dynamically configuring the endpoint, such as loading data from environment variables or configuration files, LoggerJSON provides an :on_init option that allows developers to set a module, function and list of arguments that is invoked when the endpoint starts.

config :logger_json, :backend,
  on_init: {YourApp.Logger, :load_from_system_env, []}

Encoders support

You can replace default Jason encoder with other module that supports encode_to_iodata!/1 function and encoding fragments.

Documentation

The docs can be found at https://hexdocs.pm/logger_json

Thanks

Many source code has been taken from original Elixir Logger :console back-end source code, so I want to thank all it's authors and contributors.

Part of LoggerJSON.Plug module have origins from plug_logger_json by @bleacherreport, originally licensed under Apache License 2.0. Part of LoggerJSON.PlugTest are from Elixir's Plug licensed under Apache 2.

Copyright and License

Copyright (c) 2016 Nebo #15

Released under the MIT License, which can be found in LICENSE.md.