Skip to content

Commit

Permalink
Add dns processor documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
andrewkroh committed Aug 17, 2018
1 parent a80bcc8 commit 2e56736
Showing 1 changed file with 104 additions and 0 deletions.
104 changes: 104 additions & 0 deletions libbeat/docs/processors-using.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ The supported processors are:
* <<add-docker-metadata,`add_docker_metadata`>>
* <<add-host-metadata,`add_host_metadata`>>
* <<dissect, `dissect`>>
* <<processor-dns, `dns`>>

[[conditions]]
==== Conditions
Expand Down Expand Up @@ -930,3 +931,106 @@ NOTE: A key can contain any characters except reserved suffix or prefix modifier
and `?`.

See <<conditions>> for a list of supported conditions.

[[processor-dns]]
=== DNS Reverse Lookup

The DNS processor performs reverse DNS lookups of IP addresses. It caches the
responses that it receives in accordance to the time-to-live (TTL) value
contained in the response. It also caches failures that occur during lookups.
Each instance of this processor maintains its own independent cache.

The processor uses its own DNS resolver to send requests to nameservers and does
not use the operating system's resolver. It does not read any values contained
in `/etc/hosts`.

This processor can significantly slow down your pipeline's throughput if you
have a high latency network or slow upstream nameserver. The cache will help
with performance, but if the addresses being resolved have a high cardinality
then the cache benefits will be diminished due to the high miss ratio.

By way of example, if each DNS lookup takes 2 milliseconds, the maximum
throughput you can achieve is 500 events per second (1000 milliseconds / 2
milliseconds). If you have a high cache hit ratio then your throughput can be
higher.

This is a minimal configuration example that resolves the IP addresses contained
in two fields.

[source,yaml]
----
processors:
- dns:
type: reverse
fields:
source.ip: source.hostname
destination.ip: destination.hostname
----

Next is a configuration example showing all options.

[source,yaml]
----
processors:
- dns:
type: reverse
action: append
fields:
server.ip: server.hostname
client.ip: client.hostname
success_cache:
capacity.initial: 1000
capacity.max: 10000
failure_cache:
capacity.initial: 1000
capacity.max: 10000
ttl: 1m
nameservers: ['192.0.2.1', '203.0.113.1']
timeout: 500ms
tag_on_failure: [_dns_reverse_lookup_failed]
----

The `dns` processor has the following configuration settings:

`type`:: The type of DNS lookup to perform. The only supported type is
`reverse` which queries for a PTR record.

`action`:: This defines the behavior of the processor when the target field
already exists in the event. The options are `append` (default) and `replace`.

`fields`:: This is a mapping of source field names to target field names. The
value of the source field will be used in the DNS query and result will be
written to the target field.

`success_cache.capacity.initial`:: The initial number of items that the success
cache will be allocated to hold. When initialized the processor will allocate
the memory for this number of items. Default value is `1000`.

`success_cache.capacity.max`:: The maximum number of items that the success
cache can hold. When the maximum capacity is reached a random item is evicted.
Default value is `10000`.

`failure_cache.capacity.initial`:: The initial number of items that the failure
cache will be allocated to hold. When initialized the processor will allocate
the memory for this number of items. Default value is `1000`.

`failure_cache.capacity.max`:: The maximum number of items that the failure
cache can hold. When the maximum capacity is reached a random item is evicted.
Default value is `10000`.

`failure_cache.ttl`:: The duration for which failures are cached. Valid time
units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Default value is `1m`.

`nameservers`:: A list of nameservers to query. If there are multiple servers,
the resolver queries them in the order listed. If none are specified then it
will read the nameservers listed in `/etc/resolv.conf` once at initialization.
On Windows you must always supply at least one nameserver.

`timeout`:: The duration after which a DNS query will timeout. This is timeout
for each DNS request so if you have 2 nameservers then the total timeout will be
2 times this value. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m",
"h". Default value is `500ms`.

`tag_on_failure`:: A list of tags to add to the event when any lookup fails. The
tags are only added once even if multiple lookups fail. By default no tags are
added upon failure.

0 comments on commit 2e56736

Please sign in to comment.