Improvement: Optional Top Level Keys in a Schema and Parser Option Overhaul

[jacknagz]

Background

It is becoming increasingly common for certain data types to have inconsistent top level keys. This is also a familiar scenario during external API level changes where certain keys may or may not be present in data. An example of this would be:

type 1:
{"foo": 1, "bar": 2}

changing to

type 2:
{"foo": 1, "bar": 2, "baz": 3}

To support this behavior, a new option will be added to the log declaration in conf/logs.json.

Desired Outcome

Create a new setting in the log declaration called configuration, meant for various parser options:

setting	parser(s)	type	description
optional_top_level_keys	json	Map	Key/value pairs of optional fields with their type. If an incoming record does not have the key, a default value will be provided based on the declared type.
jsonpath	json	String	The JSONPath expression used to identify and parse nested records.
envelope_keys	json	Map	When using nested records, there may be top level keys we want to include in the record. This setting adds key/value pairs to a key called envelope in a record.
delimiter	csv, kv	String	The delimiter of fields in records.
separator	kv	String	The separator between keys and values in records.

Example

  "carbonblack:ingress.event.procstart": {
    "schema": {
      "cb_server": "string",
      "command_line": "string",
      "computer_name": "string",
      "event_type": "string",
      "expect_followon_w_md5": "boolean",
      "md5": "string",
      "parent_create_time": "integer",
      "parent_md5": "string",
      "parent_path": "string",
      "parent_process_guid": "string",
      "path": "string",
      "pid": "integer",
      "process_guid": "string",
      "sensor_id": "integer",
      "timestamp": "integer",
      "type": "string",
      "username": "string"
    },
    "parser": "json",
    "configuration": {
      "optional_top_level_keys": {
        "parent_md5": "string"
      }
    }
  }