LogWather INI Files

Basic INI

Minimal INI configuration for LogWatcher to report default metrics.

[logwatcher]
log_name_format: /path/to/log/file/access
sla_ms: 300
distinguisher: lwexample
ignore_pattern: (/healthcheck|nagios)
nologsleep: 10
notify_schedule: 60
processing_time_regex: \s([\d.]*)$
processing_time_units: ms

Example INI

Here is a more specific example in which we capture several custom key=value metrics as well as creating some calculated metrics with LogWatcher.

Log Line:

1.23.45.678 logwatcher.com  - [26/Sep/2015:15:59:16 -0700] "GET /profile HTTP/1.1" 200 2989 "http://referrer.com/restaurants" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" " [wsTime=7] [isCust=0] [ver=2] [showAds=true] [daoTimelisting=2.678338] [daoTimecgmdblisting=2.678338]  [oTime=0] [pTime=1]  [daoTimecontent=2.511219] [daoTimecgmdbcontent=2.511219] [clientIp=12.3.4.5.6]" 8

INI:

[logwatcher]
log_name_format: /path/to/log/file/access
sla_ms: 300
distinguisher: lwexample
ignore_pattern: (/healthcheck|nagios)
nologsleep: 10
notify_schedule: 60
processing_time_regex: \s([\d.]*)$
processing_time_units: ms
metrics_count: return_code isCust
metrics_ratio: return_code t_wsTime t_oTime t_pTime t_daoTimeContent t_daoTimeListing apiver
metrics_sum: bytes  t_wsTime t_oTime t_pTime t_daoTimeContent t_daoTimeListing Total_Processing_Time_v2 Total_Processing_Time_v1
metric_return_code_regex: HTTP/1.\d. (\d\d\d)[ ]
metric_isCust_regex: \[isCust=(\d)\]
metric_apiver_regex: /(v\d)[^\w]
metric_bytes_regex: HTTP/1.1.\ \d\d\d (\d+)\ \"
metric_t_oTime_regex: \[oTime=([\d.]+)\]
metric_t_pTime_regex: \[pTime=([\d.]+)\]
metric_t_daoTimeContent_regex: \[daoTimecontent=([\d.]+)\]
metric_t_daoTimeListing_regex: \[daoTimelisting=([\d.]+)\]
metric_t_wsTime_regex: \[wsTime=([\d.]+)\]
metric_Total_Processing_Time_v1_regex: .* /profile/\W.* ([\d.]*)$
metric_Total_Processing_Time_v2_regex: .*v2.* ([\d.]*)$
# user-defined calculated metrics
metrics_calc: Avg_Processing_Time_v1_ms Avg_Processing_Time_v2_ms
metric_Avg_Processing_Time_v1_ms_expression: s/Total_Processing_Time_v1 / c/apiver_NotSet
metric_Avg_Processing_Time_v2_ms_expression: s/Total_Processing_Time_v2 / c/apiver_v2

All Available INI Options

Option	Possible Values Example Values	Description
log_name_format	/app/tomcat/logs/access/access_profile /usr/local/logs/access/access_guide_%Y-%m-%d:%H:00:00	The path to the log file to parse, with some optional pattern matching capabilities.
sla_ms	300	generates metrics (exceeding_SLA and exceeding_SLA_ct) based on the value specified
nologsleep	10	how long to wait for a log to show up before checking again (seconds?)
notify_schedule	60	how often to send metrics (seconds)
plugins	kafka_logger	space delimited list of plugins that you would like to use
use_graphite	True	Enable sending metrics to a graphite server assumes you have /etc/graphite.conf containing details of your graphite server.. Otherwise you should use the -G and -g command line options
brand_regex		The regex to identify unique publishers. (related to use_brand)
distinguisher	text without spaces	This will be the beginning of the name of the metric, in the form: LW_<distinguisher>_<metric> (If there's only one instance of LogWatcher running on the server, we typically set this to be the server class.)
ignore_pattern	(/healthcheck	nagios)
metric_<metric_name>_expression	s/Total_Processing_Time_v1 / c/apiver_NotSet	Calculate a new metric, based on existing metrics. The new metric name must be added to the metrics_calc list. You must specify whether the original metrics are count or sum metrics by prefixing them with c/ or s/.
metric_<metric_name>_regex	[isCust=(\d)]	The name of a metric, and the regex to pick out the value.
metrics_calc	metric1 metric2 metric3	Space separated list of metric names that you want to calculate based on the equation in the value parameter. (See the Supported metrics section above.) After adding a metric to this list, use the metric_<metric_name>_expression option to calculate a new metric, based on existing metrics.
metrics_count	metric1 metric2 metric3	Space separated list of metric names that should be treated as counters. (See the Supported metrics section above.) After adding a metric to this list, use it with the metric_<metric_name>_regex option.
metrics_dist	metric1 metric2 metric3	Space separated list of metric names that should be treated as distribution metrics. (See the Supported metrics section above.) After adding a metric to this list, use it with the metric_<metric_name>regex, metric<metric_name>count, and metric<metric_name>_size options.
metrics_list	?	Not used yet
metrics_prime	return_code_500:0 return_code_500_ratio:0	Space separated list of metrics that should be sent once at startup. (For example, this can be used to reset counters to zero when LogWatcher restarts.)
metrics_ratio	metric1 metric2 metric3	Space separated list of metric names that should be treated as ratios. (See the Supported metrics section above.) After adding a metric to this list, use it with the metric_<metric_name>_regex option.
metrics_sum	metric1 metric2 metric3	Space separated list of metric names that should be treated as sums. (See the Supported metrics section above.) After adding a metric to this list, use it with the metric_<metric_name>_regex option.
processing_time_regex	\s([\d.]*)$	Regex to pick out the processing time. (Is this a special variable/metric?)
processing_time_units	ms or us	Processing time units: ms = milliseconds (tomcat) us = microseconds (apache)
use_brand	0 or 1	0 : Disable brand features. 1 : Enable brand features. When use_brand it set to 1, you can see lines like the following in debug output: DEBUG: Found new publisher: <something that matches your brand_regex> or DEBUG: Found new publisher: NULL_brand When enabled, generates the following metrics: LW_<distinguisher>_QPS_<whatever brand_regex matches> and LW_<distinguisher>_QPS_NULL_brand