Skip to content

Commit

Permalink
Update Journalbeat docs (#8864)
Browse files Browse the repository at this point in the history
* More updates for journalbeat

* Remove unwanted comments and extra files

* Add changes from review

* Changes from second round of reviews

* Remove backoff options

* Remove questions for reviewers

* Change file to journal

* Change default  max_backoff in docs to 20s

* More fixes from review

* Revert "Remove backoff options"

This reverts commit d57e4b3.

* Add global options back
  • Loading branch information
dedemorton authored Nov 9, 2018
1 parent 22ba375 commit 508d898
Show file tree
Hide file tree
Showing 21 changed files with 309 additions and 310 deletions.
197 changes: 156 additions & 41 deletions journalbeat/docs/config-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,70 +5,68 @@
<titleabbrev>Configure inputs</titleabbrev>
++++

IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.

By default, {beatname_uc} reads log events from the default systemd journals. To
specify other journal files, set the <<{beatname_lc}-paths,`paths`>> option in
the +{beatname_lc}.inputs+ section of the +{beatname_lc}.yml+ file.

The list of paths is a YAML array, so each path begins with a dash (-). Each
path can be a directory path (to collect events from all journals in a
directory), or a file path. For example:
the +{beatname_lc}.inputs+ section of the +{beatname_lc}.yml+ file. Each path
can be a directory path (to collect events from all journals in a directory), or
a file path. For example:

["source","sh",subs="attributes"]
----
{beatname_lc}.inputs:
- paths:
- "/dev/log"
- "/var/log/messages/my-journal-file"
- "/var/log/messages/my-journal-file.journal"
----

Within the +{beatname_lc}.inputs+ section, you can also specify options that
control the position where {beatname_uc} starts reading the journal file, and
set filters to reduce the fields that {beatname_uc} needs to process. See
<<{beatname_lc}-options>> for a list of available options.

[float]
=== Configuration examples

The following example shows how to monitor multiple journals under the
same directory. {beatname_uc} merges all journals under the directory into a
single journal and reads them. With `seek` set to `cursor`, {beatname_uc}
starts reading at the beginning of the journal, but will continue reading where
it left off after a reload or restart.

Within the configuration file, you can also specify options that control how
{beatname_uc} reads the journal files and which fields are sent to the
configured output. See <<{beatname_lc}-options>> for a list of available
options.

The following examples show how to configure {beatname_uc} for some common use
cases.

[[monitor-multiple-journals]]
.Example 1: Monitor multiple journals under the same directory
This example configures {beatname_uc} to read from multiple journals that are
stored under the same directory. {beatname_uc} merges all journals under the
directory into a single event stream and reads the events. With `seek` set to
`cursor`, {beatname_uc} starts reading at the beginning of the journal, but will
continue reading at the last known position after a reload or restart.
["source","sh",subs="attributes"]
----
{beatname_lc}.inputs:
- paths: ["/path/to/journal/directory"]
seek: cursor
----

The following examples show how to get Redis events from a Docker container that
is tagged as `redis`.

//TODO: Add a better explanation of the options.

This example uses the translated fields by Journald:

[[filter-using-field-names]]
.Example 2: Fetch log events for Redis running on Docker (uses field names from systemd)
This example configures {beatname_uc} to fetch log events for Redis running in a
Docker container. The fields are matched using field names from the systemd
journal.
["source","sh",subs="attributes"]
----
{beatname_lc}.inputs:
- paths: []
include_matches:
- "container.image.tag=redis"
- "process.name=redis"
- "CONTAINER_TAG=redis"
- "_COMM=redis"
----

This example uses the field names from the systemd journal:

[[filter-using-translated-names]]
.Example 3: Fetch log events for Redis running on Docker (uses translated field names)
This example also configures {beatname_uc} to fetch log events for Redis running
in a Docker container. However, in this example the fields are matched using the
<<translated-fields,translated field names>> provided by {beatname_uc}.
["source","sh",subs="attributes"]
----
{beatname_lc}.inputs:
- paths: []
include_matches:
- "CONTAINER_TAG=redis"
- "_COMM=redis"
- "container.image.tag=redis"
- "process.name=redis"
----

[id="{beatname_lc}-options"]
Expand All @@ -86,17 +84,134 @@ path (to collect events from all journals in a directory), or a file path. If
you specify a directory, {beatname_uc} merges all journals under the directory
into a single journal and reads them.

//QUESTION: Are globs supported? If so, I need to add more detail here.
If no paths are specified, {beatname_uc} reads from the default journal.

[float]
[id="{beatname_lc}-backoff"]
==== `backoff`

The number of seconds to wait before trying to read again from journals. The
default is 1s.

[float]
[id="{beatname_lc}-max-backoff"]
==== `max_backoff`

The maximum number of seconds to wait before attempting to read again from
journals. The default is 60s.

[float]
[id="{beatname_lc}-seek"]
==== `seek`

The position to start reading the journal from. Valid settings are:

* `head`: Starts reading at the beginning of the file.
* `tail`: Starts reading at the end of the file.
* `cursor`: Initially starts reading at the beginning of the file, but continues
reading where it left off after a reload or restart.
* `head`: Starts reading at the beginning of the journal. After a restart,
{beatname_uc} resends all log messages in the journal.
* `tail`: Starts reading at the end of the journal. After a restart,
{beatname_uc} resends the last message, which might result in duplicates. If
multiple log messages are written to a journal while {beatname_uc} is down,
only the last log message is sent on restart.
* `cursor`: On first read, starts reading at the beginning of the journal. After a
reload or restart, continues reading at the last known position.

When specified under `paths`, the `seek` setting applies to all journals under
the configured paths. When specified directly under the +{beatname_lc}+
namespace, the setting applies to all journals read by {beatname_uc}.

//TODO: ADD OTHER OPTIONS HERE.
If you have old log files and want to skip lines, start {beatname_uc} with
`seek: tail` specified. Then stop {beatname_uc}, set `seek: cursor`, and restart
{beatname_uc}.

[float]
[id="{beatname_lc}-include-matches"]
==== `include_matches`

A list of filter expressions used to match fields. The format of the expression
is `field=value`. {beatname_uc} fetches all events that exactly match the
expressions. Pattern matching is not supported.

To reference fields, use one of the following:

* The field name used by the systemd journal. For example,
`CONTAINER_TAG=redis` (<<filter-using-field-names,see a full example>>).
* The <<translated-fields,translated field name>> used by
{beatname_uc}. For example, `container.image.tag=redis`
(<<filter-using-translated-names,see a full example>>). {beatname_uc}
does not translate all fields from the journal. For custom fields, use the name
specified in the systemd journal.

When specified under `paths`, the `include_matches` filter is applied to all
journals under the configured paths. When specified directly under the
+{beatname_lc}+ namespace, the setting applies to all journals read by
{beatname_uc}.

[float]
[[translated-fields]]
=== Translated field names

You can use the following translated names in filter expressions to reference
journald fields:

[horizontal]
*Journald field name*:: *Translated name*
`COREDUMP_UNIT`:: `journald.coredump.unit`
`COREDUMP_USER_UNIT`:: `journald.coredump.user_unit`
`OBJECT_AUDIT_LOGINUID`:: `journald.object.audit.login_uid`
`OBJECT_AUDIT_SESSION`:: `journald.object.audit.session`
`OBJECT_CMDLINE`:: `journald.object.cmd`
`OBJECT_COMM`:: `journald.object.name`
`OBJECT_EXE`:: `journald.object.executable`
`OBJECT_GID`:: `journald.object.gid`
`OBJECT_PID`:: `journald.object.pid`
`OBJECT_SYSTEMD_OWNER_UID`:: `journald.object.systemd.owner_uid`
`OBJECT_SYSTEMD_SESSION`:: `journald.object.systemd.session`
`OBJECT_SYSTEMD_UNIT`:: `journald.object.systemd.unit`
`OBJECT_SYSTEMD_USER_UNIT`:: `journald.object.systemd.user_unit`
`OBJECT_UID`:: `journald.object.uid`
`_AUDIT_LOGINUID`:: `process.audit.login_uid`
`_AUDIT_SESSION`:: `process.audit.session`
`_BOOT_ID`:: `host.boot_id`
`_CAP_EFFECTIVE`:: `process.capabilites`
`_CMDLINE`:: `process.cmd`
`_CODE_FILE`:: `journald.code.file`
`_CODE_FUNC`:: `journald.code.func`
`_CODE_LINE`:: `journald.code.line`
`_COMM`:: `process.name`
`_EXE`:: `process.executable`
`_GID`:: `process.uid`
`_HOSTNAME`:: `host.name`
`_KERNEL_DEVICE`:: `journald.kernel.device`
`_KERNEL_SUBSYSTEM`:: `journald.kernel.subsystem`
`_MACHINE_ID`:: `host.id`
`_MESSAGE`:: `message`
`_PID`:: `process.pid`
`_PRIORITY`:: `syslog.priority`
`_SYSLOG_FACILITY`:: `syslog.facility`
`_SYSLOG_IDENTIFIER`:: `syslog.identifier`
`_SYSLOG_PID`:: `syslog.pid`
`_SYSTEMD_CGROUP`:: `systemd.cgroup`
`_SYSTEMD_INVOCATION_ID`:: `systemd.invocation_id`
`_SYSTEMD_OWNER_UID`:: `systemd.owner_uid`
`_SYSTEMD_SESSION`:: `systemd.session`
`_SYSTEMD_SLICE`:: `systemd.slice`
`_SYSTEMD_UNIT`:: `systemd.unit`
`_SYSTEMD_USER_SLICE`:: `systemd.user_slice`
`_SYSTEMD_USER_UNIT`:: `systemd.user_unit`
`_TRANSPORT`:: `systemd.transport`
`_UDEV_DEVLINK`:: `journald.kernel.device_symlinks`
`_UDEV_DEVNODE`:: `journald.kernel.device_node_path`
`_UDEV_SYSNAME`:: `journald.kernel.device_name`
`_UID`:: `process.uid`


The following translated fields for
https://docs.docker.com/config/containers/logging/journald/[Docker] are also
available:

[horizontal]
`CONTAINER_ID`:: `conatiner.id_truncated`
`CONTAINER_ID_FULL`:: `container.id`
`CONTAINER_NAME`:: `container.name`
`CONTAINER_PARTIAL_MESSAGE`:: `container.partial`
`CONTAINER_TAG`:: `container.image.tag`
24 changes: 3 additions & 21 deletions journalbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,11 @@
[partintro]
--

IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.

Before modifying configuration settings, make sure you've completed the
<<{beatname_lc}-configuration,configuration steps>> in the Getting Started.
This section describes some common use cases for changing configuration options.

To configure {beatname_uc}, you edit the configuration file. For rpm and deb,
you’ll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+.
There's also a full example configuration file at
+/etc/{beatname_lc}/{beatname_lc}.reference.yml+ that shows all non-deprecated
options. For mac and win, look in the archive that you extracted.

The {beatname_uc} configuration file uses http://yaml.org/[YAML] for its syntax.
See the {libbeat}/config-file-format.html[Config File Format] section of the
_{libbeat_docs}_ for more about the structure of the config file.
include::../../libbeat/docs/shared-configuring.asciidoc[]

The following topics describe how to configure {beatname_uc}:

Expand All @@ -31,15 +21,13 @@ The following topics describe how to configure {beatname_uc}:
* <<configuring-ingest-node>>
* <<configuration-path>>
* <<setup-kibana-endpoint>>
* <<configuration-dashboards>>
* <<configuration-template>>
* <<configuration-logging>>
* <<using-environ-vars>>
//* <<configuration-autodiscover>>
* <<yaml-tips>>
* <<regexp-support>>
* <<http-endpoint>>
//* <<{beatname_lc}-reference-yml>>
* <<{beatname_lc}-reference-yml>>

--

Expand Down Expand Up @@ -69,9 +57,6 @@ include::../../libbeat/docs/loggingconfig.asciidoc[]
include::../../libbeat/docs/shared-env-vars.asciidoc[]
:standalone!:

//OPEN ISSUE: DO WE NEED AUTODISCOVER?
//include::../../libbeat/docs/shared-autodiscover.asciidoc[]

:standalone:
include::../../libbeat/docs/yaml.asciidoc[]
:standalone!:
Expand All @@ -80,7 +65,4 @@ include::../../libbeat/docs/regexp.asciidoc[]

include::../../libbeat/docs/http-endpoint.asciidoc[]

// TODO: Uncomment the following include statement when the reference yaml file
// is available in the repo. Also uncomment the link in the jump list at the top
// of this file.
//include::../../libbeat/docs/reference-yml.asciidoc[]
include::../../libbeat/docs/reference-yml.asciidoc[]
14 changes: 0 additions & 14 deletions journalbeat/docs/faq.asciidoc
Original file line number Diff line number Diff line change
@@ -1,24 +1,10 @@
[[faq]]
== Frequently asked questions

IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.

This section contains frequently asked questions about {beatname_uc}. Also check
out the https://discuss.elastic.co/c/beats/{beatname_lc}[{beatname_uc}
discussion forum].

[float]
[id="{beatname_lc}-sometext"]
=== Question 1?

ADD DESCRIPTION HERE

[float]
[id="{beatname_lc}-sometext2"]
=== Question 2?

ADD DESCRIPTION HERE

include::../../libbeat/docs/faq-limit-bandwidth.asciidoc[]

include::../../libbeat/docs/shared-faq.asciidoc[]
19 changes: 4 additions & 15 deletions journalbeat/docs/filtering.asciidoc
Original file line number Diff line number Diff line change
@@ -1,20 +1,15 @@
[[filtering-and-enhancing-data]]
== Filter and enhance the exported data

IMPORTANT: This documentation is placeholder content. It has not yet been reviewed.

Your use case might require only a subset of the data exported by {beatname_uc},
or you might need to enhance the exported data (for example, by adding
metadata). {beatname_uc} provides a couple of options for filtering and
enhancing exported data.

You can configure each input to include or exclude specific lines or files. This
allows you to specify different filtering criteria for each input. To do this,
you use the `include_lines`, `exclude_lines`, and `exclude_files` options under
the +{beatname_lc}.inputs+ section of the config file (see
<<configuration-{beatname_lc}-options>>). The disadvantage of this approach is
that you need to implement a configuration option for each filtering criteria
that you need.
You can configure {beatname_uc} to include events that match specific filtering
criteria. To do this, use the <<{beatname_lc}-include-matches,`include_matches`>>
option. The advantage of this approach is that you can reduce the number of
fields that {beatname_uc} needs to process.

Another approach (the one described here) is to define processors to configure
global processing across all data exported by {beatname_uc}.
Expand All @@ -26,12 +21,6 @@ global processing across all data exported by {beatname_uc}.

include::../../libbeat/docs/processors.asciidoc[]

[float]
[[specific-example]]
==== XYZ example

ADD EXAMPLES SPECIFIC TO THE BEAT, OR DELETE THIS SECTION

// You must set the processor-scope attribute to resolve the attribute reference
// defined in processors-using.asciidoc. The attribute is used to indicate where
// processors are valid. If processors are valid in more than two locations
Expand Down
Loading

0 comments on commit 508d898

Please sign in to comment.