From ba58e4fc348377e04faac02f6dcd87f3e54451c6 Mon Sep 17 00:00:00 2001 From: DeDe Morton Date: Fri, 9 Nov 2018 10:55:21 -0800 Subject: [PATCH] Update Journalbeat docs (#8864) * More updates for journalbeat * Remove unwanted comments and extra files * Add changes from review * Changes from second round of reviews * Remove backoff options * Remove questions for reviewers * Change file to journal * Change default max_backoff in docs to 20s * More fixes from review * Revert "Remove backoff options" This reverts commit d57e4b32f125e2b10794930678de7077f1f31981. * Add global options back --- journalbeat/docs/config-options.asciidoc | 197 ++++++++++++++---- journalbeat/docs/configuring-howto.asciidoc | 24 +-- journalbeat/docs/faq.asciidoc | 14 -- journalbeat/docs/filtering.asciidoc | 19 +- journalbeat/docs/general-options.asciidoc | 54 ++++- journalbeat/docs/getting-started.asciidoc | 145 ++++--------- journalbeat/docs/how-it-works.asciidoc | 6 - journalbeat/docs/index.asciidoc | 5 +- journalbeat/docs/overview.asciidoc | 2 - .../docs/running-on-kubernetes.asciidoc | 6 - journalbeat/docs/setting-up-running.asciidoc | 13 -- journalbeat/docs/troubleshooting.asciidoc | 2 - libbeat/docs/command-reference.asciidoc | 16 +- libbeat/docs/outputconfig.asciidoc | 5 +- libbeat/docs/security/securing-beats.asciidoc | 4 + libbeat/docs/setup-config.asciidoc | 4 +- libbeat/docs/shared-configuring.asciidoc | 15 +- .../docs/shared-download-and-install.asciidoc | 17 +- .../shared-getting-started-intro.asciidoc | 2 +- libbeat/docs/shared-template-load.asciidoc | 63 ++---- .../docs/step-configure-credentials.asciidoc | 6 +- 21 files changed, 309 insertions(+), 310 deletions(-) delete mode 100644 journalbeat/docs/how-it-works.asciidoc delete mode 100644 journalbeat/docs/running-on-kubernetes.asciidoc diff --git a/journalbeat/docs/config-options.asciidoc b/journalbeat/docs/config-options.asciidoc index 722d9b9534a..4cfdb7d9e09 100644 --- a/journalbeat/docs/config-options.asciidoc +++ b/journalbeat/docs/config-options.asciidoc @@ -5,38 +5,35 @@ Configure inputs ++++ -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - By default, {beatname_uc} reads log events from the default systemd journals. To specify other journal files, set the <<{beatname_lc}-paths,`paths`>> option in -the +{beatname_lc}.inputs+ section of the +{beatname_lc}.yml+ file. - -The list of paths is a YAML array, so each path begins with a dash (-). Each -path can be a directory path (to collect events from all journals in a -directory), or a file path. For example: +the +{beatname_lc}.inputs+ section of the +{beatname_lc}.yml+ file. Each path +can be a directory path (to collect events from all journals in a directory), or +a file path. For example: ["source","sh",subs="attributes"] ---- {beatname_lc}.inputs: - paths: - "/dev/log" - - "/var/log/messages/my-journal-file" + - "/var/log/messages/my-journal-file.journal" ---- -Within the +{beatname_lc}.inputs+ section, you can also specify options that -control the position where {beatname_uc} starts reading the journal file, and -set filters to reduce the fields that {beatname_uc} needs to process. See -<<{beatname_lc}-options>> for a list of available options. - -[float] -=== Configuration examples - -The following example shows how to monitor multiple journals under the -same directory. {beatname_uc} merges all journals under the directory into a -single journal and reads them. With `seek` set to `cursor`, {beatname_uc} -starts reading at the beginning of the journal, but will continue reading where -it left off after a reload or restart. - +Within the configuration file, you can also specify options that control how +{beatname_uc} reads the journal files and which fields are sent to the +configured output. See <<{beatname_lc}-options>> for a list of available +options. + +The following examples show how to configure {beatname_uc} for some common use +cases. + +[[monitor-multiple-journals]] +.Example 1: Monitor multiple journals under the same directory +This example configures {beatname_uc} to read from multiple journals that are +stored under the same directory. {beatname_uc} merges all journals under the +directory into a single event stream and reads the events. With `seek` set to +`cursor`, {beatname_uc} starts reading at the beginning of the journal, but will +continue reading at the last known position after a reload or restart. ["source","sh",subs="attributes"] ---- {beatname_lc}.inputs: @@ -44,31 +41,32 @@ it left off after a reload or restart. seek: cursor ---- -The following examples show how to get Redis events from a Docker container that -is tagged as `redis`. - -//TODO: Add a better explanation of the options. - -This example uses the translated fields by Journald: - +[[filter-using-field-names]] +.Example 2: Fetch log events for Redis running on Docker (uses field names from systemd) +This example configures {beatname_uc} to fetch log events for Redis running in a +Docker container. The fields are matched using field names from the systemd +journal. ["source","sh",subs="attributes"] ---- {beatname_lc}.inputs: - paths: [] include_matches: - - "container.image.tag=redis" - - "process.name=redis" + - "CONTAINER_TAG=redis" + - "_COMM=redis" ---- -This example uses the field names from the systemd journal: - +[[filter-using-translated-names]] +.Example 3: Fetch log events for Redis running on Docker (uses translated field names) +This example also configures {beatname_uc} to fetch log events for Redis running +in a Docker container. However, in this example the fields are matched using the +<> provided by {beatname_uc}. ["source","sh",subs="attributes"] ---- {beatname_lc}.inputs: - paths: [] include_matches: - - "CONTAINER_TAG=redis" - - "_COMM=redis" + - "container.image.tag=redis" + - "process.name=redis" ---- [id="{beatname_lc}-options"] @@ -86,7 +84,21 @@ path (to collect events from all journals in a directory), or a file path. If you specify a directory, {beatname_uc} merges all journals under the directory into a single journal and reads them. -//QUESTION: Are globs supported? If so, I need to add more detail here. +If no paths are specified, {beatname_uc} reads from the default journal. + +[float] +[id="{beatname_lc}-backoff"] +==== `backoff` + +The number of seconds to wait before trying to read again from journals. The +default is 1s. + +[float] +[id="{beatname_lc}-max-backoff"] +==== `max_backoff` + +The maximum number of seconds to wait before attempting to read again from +journals. The default is 60s. [float] [id="{beatname_lc}-seek"] @@ -94,9 +106,112 @@ into a single journal and reads them. The position to start reading the journal from. Valid settings are: -* `head`: Starts reading at the beginning of the file. -* `tail`: Starts reading at the end of the file. -* `cursor`: Initially starts reading at the beginning of the file, but continues -reading where it left off after a reload or restart. +* `head`: Starts reading at the beginning of the journal. After a restart, +{beatname_uc} resends all log messages in the journal. +* `tail`: Starts reading at the end of the journal. After a restart, +{beatname_uc} resends the last message, which might result in duplicates. If +multiple log messages are written to a journal while {beatname_uc} is down, +only the last log message is sent on restart. +* `cursor`: On first read, starts reading at the beginning of the journal. After a +reload or restart, continues reading at the last known position. + +When specified under `paths`, the `seek` setting applies to all journals under +the configured paths. When specified directly under the +{beatname_lc}+ +namespace, the setting applies to all journals read by {beatname_uc}. -//TODO: ADD OTHER OPTIONS HERE. \ No newline at end of file +If you have old log files and want to skip lines, start {beatname_uc} with +`seek: tail` specified. Then stop {beatname_uc}, set `seek: cursor`, and restart +{beatname_uc}. + +[float] +[id="{beatname_lc}-include-matches"] +==== `include_matches` + +A list of filter expressions used to match fields. The format of the expression +is `field=value`. {beatname_uc} fetches all events that exactly match the +expressions. Pattern matching is not supported. + +To reference fields, use one of the following: + +* The field name used by the systemd journal. For example, +`CONTAINER_TAG=redis` (<>). +* The <> used by +{beatname_uc}. For example, `container.image.tag=redis` +(<>). {beatname_uc} +does not translate all fields from the journal. For custom fields, use the name +specified in the systemd journal. + +When specified under `paths`, the `include_matches` filter is applied to all +journals under the configured paths. When specified directly under the ++{beatname_lc}+ namespace, the setting applies to all journals read by +{beatname_uc}. + +[float] +[[translated-fields]] +=== Translated field names + +You can use the following translated names in filter expressions to reference +journald fields: + +[horizontal] +*Journald field name*:: *Translated name* +`COREDUMP_UNIT`:: `journald.coredump.unit` +`COREDUMP_USER_UNIT`:: `journald.coredump.user_unit` +`OBJECT_AUDIT_LOGINUID`:: `journald.object.audit.login_uid` +`OBJECT_AUDIT_SESSION`:: `journald.object.audit.session` +`OBJECT_CMDLINE`:: `journald.object.cmd` +`OBJECT_COMM`:: `journald.object.name` +`OBJECT_EXE`:: `journald.object.executable` +`OBJECT_GID`:: `journald.object.gid` +`OBJECT_PID`:: `journald.object.pid` +`OBJECT_SYSTEMD_OWNER_UID`:: `journald.object.systemd.owner_uid` +`OBJECT_SYSTEMD_SESSION`:: `journald.object.systemd.session` +`OBJECT_SYSTEMD_UNIT`:: `journald.object.systemd.unit` +`OBJECT_SYSTEMD_USER_UNIT`:: `journald.object.systemd.user_unit` +`OBJECT_UID`:: `journald.object.uid` +`_AUDIT_LOGINUID`:: `process.audit.login_uid` +`_AUDIT_SESSION`:: `process.audit.session` +`_BOOT_ID`:: `host.boot_id` +`_CAP_EFFECTIVE`:: `process.capabilites` +`_CMDLINE`:: `process.cmd` +`_CODE_FILE`:: `journald.code.file` +`_CODE_FUNC`:: `journald.code.func` +`_CODE_LINE`:: `journald.code.line` +`_COMM`:: `process.name` +`_EXE`:: `process.executable` +`_GID`:: `process.uid` +`_HOSTNAME`:: `host.name` +`_KERNEL_DEVICE`:: `journald.kernel.device` +`_KERNEL_SUBSYSTEM`:: `journald.kernel.subsystem` +`_MACHINE_ID`:: `host.id` +`_MESSAGE`:: `message` +`_PID`:: `process.pid` +`_PRIORITY`:: `syslog.priority` +`_SYSLOG_FACILITY`:: `syslog.facility` +`_SYSLOG_IDENTIFIER`:: `syslog.identifier` +`_SYSLOG_PID`:: `syslog.pid` +`_SYSTEMD_CGROUP`:: `systemd.cgroup` +`_SYSTEMD_INVOCATION_ID`:: `systemd.invocation_id` +`_SYSTEMD_OWNER_UID`:: `systemd.owner_uid` +`_SYSTEMD_SESSION`:: `systemd.session` +`_SYSTEMD_SLICE`:: `systemd.slice` +`_SYSTEMD_UNIT`:: `systemd.unit` +`_SYSTEMD_USER_SLICE`:: `systemd.user_slice` +`_SYSTEMD_USER_UNIT`:: `systemd.user_unit` +`_TRANSPORT`:: `systemd.transport` +`_UDEV_DEVLINK`:: `journald.kernel.device_symlinks` +`_UDEV_DEVNODE`:: `journald.kernel.device_node_path` +`_UDEV_SYSNAME`:: `journald.kernel.device_name` +`_UID`:: `process.uid` + + +The following translated fields for +https://docs.docker.com/config/containers/logging/journald/[Docker] are also +available: + +[horizontal] +`CONTAINER_ID`:: `conatiner.id_truncated` +`CONTAINER_ID_FULL`:: `container.id` +`CONTAINER_NAME`:: `container.name` +`CONTAINER_PARTIAL_MESSAGE`:: `container.partial` +`CONTAINER_TAG`:: `container.image.tag` diff --git a/journalbeat/docs/configuring-howto.asciidoc b/journalbeat/docs/configuring-howto.asciidoc index be66d268251..c23d40d50e2 100644 --- a/journalbeat/docs/configuring-howto.asciidoc +++ b/journalbeat/docs/configuring-howto.asciidoc @@ -4,21 +4,11 @@ [partintro] -- -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - Before modifying configuration settings, make sure you've completed the <<{beatname_lc}-configuration,configuration steps>> in the Getting Started. This section describes some common use cases for changing configuration options. -To configure {beatname_uc}, you edit the configuration file. For rpm and deb, -you’ll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+. -There's also a full example configuration file at -+/etc/{beatname_lc}/{beatname_lc}.reference.yml+ that shows all non-deprecated -options. For mac and win, look in the archive that you extracted. - -The {beatname_uc} configuration file uses http://yaml.org/[YAML] for its syntax. -See the {libbeat}/config-file-format.html[Config File Format] section of the -_{libbeat_docs}_ for more about the structure of the config file. +include::../../libbeat/docs/shared-configuring.asciidoc[] The following topics describe how to configure {beatname_uc}: @@ -31,15 +21,13 @@ The following topics describe how to configure {beatname_uc}: * <> * <> * <> -* <> * <> * <> * <> -//* <> * <> * <> * <> -//* <<{beatname_lc}-reference-yml>> +* <<{beatname_lc}-reference-yml>> -- @@ -69,9 +57,6 @@ include::../../libbeat/docs/loggingconfig.asciidoc[] include::../../libbeat/docs/shared-env-vars.asciidoc[] :standalone!: -//OPEN ISSUE: DO WE NEED AUTODISCOVER? -//include::../../libbeat/docs/shared-autodiscover.asciidoc[] - :standalone: include::../../libbeat/docs/yaml.asciidoc[] :standalone!: @@ -80,7 +65,4 @@ include::../../libbeat/docs/regexp.asciidoc[] include::../../libbeat/docs/http-endpoint.asciidoc[] -// TODO: Uncomment the following include statement when the reference yaml file -// is available in the repo. Also uncomment the link in the jump list at the top -// of this file. -//include::../../libbeat/docs/reference-yml.asciidoc[] +include::../../libbeat/docs/reference-yml.asciidoc[] diff --git a/journalbeat/docs/faq.asciidoc b/journalbeat/docs/faq.asciidoc index 6d5b4ed296e..9e0d0a0c158 100644 --- a/journalbeat/docs/faq.asciidoc +++ b/journalbeat/docs/faq.asciidoc @@ -1,24 +1,10 @@ [[faq]] == Frequently asked questions -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - This section contains frequently asked questions about {beatname_uc}. Also check out the https://discuss.elastic.co/c/beats/{beatname_lc}[{beatname_uc} discussion forum]. -[float] -[id="{beatname_lc}-sometext"] -=== Question 1? - -ADD DESCRIPTION HERE - -[float] -[id="{beatname_lc}-sometext2"] -=== Question 2? - -ADD DESCRIPTION HERE - include::../../libbeat/docs/faq-limit-bandwidth.asciidoc[] include::../../libbeat/docs/shared-faq.asciidoc[] diff --git a/journalbeat/docs/filtering.asciidoc b/journalbeat/docs/filtering.asciidoc index e75b4e73bd7..c9182ebe46d 100644 --- a/journalbeat/docs/filtering.asciidoc +++ b/journalbeat/docs/filtering.asciidoc @@ -1,20 +1,15 @@ [[filtering-and-enhancing-data]] == Filter and enhance the exported data -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - Your use case might require only a subset of the data exported by {beatname_uc}, or you might need to enhance the exported data (for example, by adding metadata). {beatname_uc} provides a couple of options for filtering and enhancing exported data. -You can configure each input to include or exclude specific lines or files. This -allows you to specify different filtering criteria for each input. To do this, -you use the `include_lines`, `exclude_lines`, and `exclude_files` options under -the +{beatname_lc}.inputs+ section of the config file (see -<>). The disadvantage of this approach is -that you need to implement a configuration option for each filtering criteria -that you need. +You can configure {beatname_uc} to include events that match specific filtering +criteria. To do this, use the <<{beatname_lc}-include-matches,`include_matches`>> +option. The advantage of this approach is that you can reduce the number of +fields that {beatname_uc} needs to process. Another approach (the one described here) is to define processors to configure global processing across all data exported by {beatname_uc}. @@ -26,12 +21,6 @@ global processing across all data exported by {beatname_uc}. include::../../libbeat/docs/processors.asciidoc[] -[float] -[[specific-example]] -==== XYZ example - -ADD EXAMPLES SPECIFIC TO THE BEAT, OR DELETE THIS SECTION - // You must set the processor-scope attribute to resolve the attribute reference // defined in processors-using.asciidoc. The attribute is used to indicate where // processors are valid. If processors are valid in more than two locations diff --git a/journalbeat/docs/general-options.asciidoc b/journalbeat/docs/general-options.asciidoc index 97367b71aac..71ab82fc54c 100644 --- a/journalbeat/docs/general-options.asciidoc +++ b/journalbeat/docs/general-options.asciidoc @@ -1,10 +1,58 @@ [[configuration-general-options]] == Specify general settings -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - You can specify settings in the +{beatname_lc}.yml+ config file to control the -general behavior of {beatname_uc}. +general behavior of {beatname_uc}. This includes: + +* <> that control things like +publisher behavior and the location of some files. + +* <> that are supported by all Elastic +Beats. + +[float] +[[configuration-global-options]] +=== Global {beatname_uc} configuration options + +These options are in the +{beatname_lc}+ namespace. + +[float] +[id="{beatname_lc}-registry-file"] +==== `registry_file` + +The name of the registry file. If a relative path is used, it is considered relative to the +data path. See the <> section for details. The default is `${path.data}/registry`. + +["source","sh",subs="attributes"] +---- +{beatname_lc}.registry_file: registry +---- + +[float] +==== `backoff` +This option is valid as a global setting under the +{beatname_lc}+ namespace +or under `paths`. For a description of this option, see +<<{beatname_lc}-backoff,`backoff`>>. + +[float] +==== `max_backoff` +This option is valid as a global setting under the +{beatname_lc}+ namespace +or under `paths`. For a description of this option, see +<<{beatname_lc}-max-backoff,`max_backoff`>>. + +[float] +==== `seek` + +This option is valid as a global setting under the +{beatname_lc}+ namespace +or under `paths`. For a description of this option, see +<<{beatname_lc}-seek,`seek`>>. + +[float] +==== `include_matches` + +This option is valid as a global setting under the +{beatname_lc}+ namespace +or under `paths`. For a description of this option, see +<<{beatname_lc}-include-matches,`include_matches`>>. include::../../libbeat/docs/generalconfig.asciidoc[] diff --git a/journalbeat/docs/getting-started.asciidoc b/journalbeat/docs/getting-started.asciidoc index 2be942e9701..37b835b96d0 100644 --- a/journalbeat/docs/getting-started.asciidoc +++ b/journalbeat/docs/getting-started.asciidoc @@ -1,24 +1,17 @@ [id="{beatname_lc}-getting-started"] -== Getting Started With {beatname_uc} - -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. +== Getting started with {beatname_uc} include::../../libbeat/docs/shared-getting-started-intro.asciidoc[] * <<{beatname_lc}-installation>> * <<{beatname_lc}-configuration>> * <<{beatname_lc}-template>> -* <> * <<{beatname_lc}-starting>> -* <> * <> [id="{beatname_lc}-installation"] === Step 1: Install {beatname_uc} -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - -:no-docker: include::../../libbeat/docs/shared-download-and-install.asciidoc[] [[deb]] @@ -59,8 +52,8 @@ sudo rpm -vi {beatname_lc}-{version}-x86_64.rpm endif::[] -[[mac]] -*mac:* +[[linux]] +*linux:* ifeval::["{release-state}"=="unreleased"] @@ -72,51 +65,19 @@ ifeval::["{release-state}"!="unreleased"] ["source","sh",subs="attributes"] ------------------------------------------------ -curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-darwin-x86_64.tar.gz -tar xzvf {beatname_lc}-{version}-darwin-x86_64.tar.gz +curl -L -O https://artifacts.elastic.co/downloads/beats/{beatname_lc}/{beatname_lc}-{version}-linux-x86_64.tar.gz +tar xzvf {beatname_lc}-{version}-linux-x86_64.tar.gz ------------------------------------------------ endif::[] -[[win]] -*win:* - -ifeval::["{release-state}"=="unreleased"] - -Version {version} of {beatname_uc} has not yet been released. - -endif::[] - -ifeval::["{release-state}"!="unreleased"] - -. Download the {beatname_uc} Windows zip file from the -https://www.elastic.co/downloads/beats/{beatname_lc}[downloads page]. - -. Extract the contents of the zip file into `C:\Program Files`. - -. Rename the +{beatname_lc}--windows+ directory to +{beatname_uc}+. - -. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*). - -. From the PowerShell prompt, run the following commands to install {beatname_uc} as a -Windows service: -+ -["source","sh",subs="attributes"] ----------------------------------------------------------------------- -PS > cd 'C:{backslash}Program Files{backslash}{beatname_uc}' -PS C:{backslash}Program Files{backslash}{beatname_uc}> .{backslash}install-service-{beatname_lc}.ps1 ----------------------------------------------------------------------- - -NOTE: If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: +PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-{beatname_lc}.ps1+. - -endif::[] - [id="{beatname_lc}-configuration"] === Step 2: Configure {beatname_uc} -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. +Before running {beatname_uc}, you can specify the location of the systemd +journal files and configure how you want the files to be read. If you accept the +default configuration, {beatname_uc} reads from the local journal. -:no-docker: include::../../libbeat/docs/shared-configuring.asciidoc[] Here is a sample of the +{beatname_lc}+ section of the +{beatname_lc}.yml+ file. @@ -126,7 +87,7 @@ Here is a sample of the +{beatname_lc}+ section of the +{beatname_lc}.yml+ file. ---------------------------------------------------------------------- journalbeat.inputs: - paths: ["/path/to/journal/directory"] - seek: cursor + seek: head ---------------------------------------------------------------------- To configure {beatname_uc}: @@ -140,20 +101,23 @@ path. For example: {beatname_lc}.inputs: - paths: - "/dev/log" - - "/var/log/messages/my-journal-file" + - "/var/log/messages/my-journal-file.journal" ---- + If no paths are specified, {beatname_uc} reads from the default journal. -. Set the `seek` option to control the position where {beatname_uc} starts -reading the journal. The available options are `head`, `tail`, and `cursor`. -Typically, you'll set `seek: cursor` so {beatname_uc} can continue reading -where it left off after a reload or restart. - -. Optional: Set the `include_matches` option to filter entries in journald -before collecting any log events. This reduces the number of fields that the -Beat needs to process. For example, to fetch only Redis events from a Docker -container tagged as `redis`, use: +. Set the <<{beatname_lc}-seek,`seek`>> option to control the position where +{beatname_uc} starts reading the journal. The available options are `head`, +`tail`, and `cursor`. The default is `cursor`, which means that on first read, +{beatname_uc} starts reading at the beginning of the file, but continues reading +at the last known position after a reload or restart. For more detail about +the settings, see the reference docs for the +<<{beatname_lc}-seek,`seek` option>>. + +. (Optional) Set the <<{beatname_lc}-include-matches,`include_matches`>> option +to filter entries in journald before collecting any log events. This reduces the +number of events that {beatname_uc} needs to process. For example, to fetch only +Redis events from a Docker container tagged as `redis`, use: + ["source","sh",subs="attributes"] ---- @@ -163,8 +127,6 @@ container tagged as `redis`, use: - "CONTAINER_TAG=redis" - "_COMM=redis" ---- -+ -See <> for more about this setting. include::../../libbeat/docs/step-configure-output.asciidoc[] @@ -180,21 +142,10 @@ include::../../libbeat/docs/step-look-at-config.asciidoc[] [id="{beatname_lc}-template"] === Step 3: Load the index template in Elasticsearch -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - include::../../libbeat/docs/shared-template-load.asciidoc[] -[[load-kibana-dashboards]] -=== Step 4: Set up the Kibana dashboards - -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - -include::../../libbeat/docs/dashboards.asciidoc[] - [id="{beatname_lc}-starting"] -=== Step 5: Start {beatname_uc} - -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. +=== Step 4: Start {beatname_uc} Start {beatname_uc} by issuing the appropriate command for your platform. If you are accessing a secured Elasticsearch cluster, make sure you've configured @@ -211,12 +162,12 @@ start {beatname_uc} in the foreground. sudo service {beatname_lc} start ---------------------------------------------------------------------- -*mac:* +*linux:* ["source","sh",subs="attributes"] ---------------------------------------------------------------------- sudo chown root {beatname_lc}.yml <1> -sudo ./{beatname_lc} -e -c {beatname_lc}.yml -d "publish" +sudo ./{beatname_lc} -e ---------------------------------------------------------------------- <1> You'll be running {beatname_uc} as root, so you need to change ownership of the configuration file, or run {beatname_uc} with `--strict.perms=false` @@ -224,38 +175,28 @@ specified. See {libbeat}/config-file-permissions.html[Config File Ownership and Permissions] in the _Beats Platform Reference_. -*win:* - -["source","sh",subs="attributes"] ----------------------------------------------------------------------- -PS C:{backslash}Program Files{backslash}{beatname_uc}> Start-Service {beatname_lc} ----------------------------------------------------------------------- - - -By default, Windows log files are stored in +C:\ProgramData\{beatname_lc}\Logs+. - -{beatname_uc} is now ready to send log files to your defined output. +{beatname_uc} is now ready to send journal events to the defined output. [[view-kibana-dashboards]] -=== Step 6: View the sample Kibana dashboards - -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - -To make it easier for you to explore {beatname_uc} data in Kibana, we've created -example {beatname_uc} dashboards. You loaded the dashboards earlier when you -ran the `setup` command. +=== Step 5: View your data in Kibana -include::../../libbeat/docs/opendashboards.asciidoc[] +There are currently no example dashboards available for {beatname_uc}. -You can use these dashboards as examples and -{kibana-ref}/dashboard.html[customize] them to meet your needs. +To learn how to view and explore your data, see the +_{kibana-ref}/index.html[{kib} User Guide]_. -To populate the example dashboards with data, you need to either -<> or use Logstash to -parse the data into the fields expected by the dashboards. +[NOTE] +===== +By default, the Logs UI in {kib} only shows logs from `filebeat-*` +indexes. To show {beatname_uc} indexes, add the following settings to the {kib} +configuration: -Here is an example of the {beatname_uc} ADD DASHBOARD NAME dashboard: +[source,yaml] +---- +xpack.infra: + sources: + default: + logAlias: "filebeat-*,journalbeat-*" +---- -// Add an example of the dashboard -//[role="screenshot"] -//image:./images/add-image-name.png[] +===== diff --git a/journalbeat/docs/how-it-works.asciidoc b/journalbeat/docs/how-it-works.asciidoc deleted file mode 100644 index 375c55507e3..00000000000 --- a/journalbeat/docs/how-it-works.asciidoc +++ /dev/null @@ -1,6 +0,0 @@ -[id="how-{beatname_lc}-works"] -== How {beatname_uc} works - -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - -DESCRIBE HOW THE BEAT WORKS. diff --git a/journalbeat/docs/index.asciidoc b/journalbeat/docs/index.asciidoc index 6bb3fdecbcc..2c6febb6417 100644 --- a/journalbeat/docs/index.asciidoc +++ b/journalbeat/docs/index.asciidoc @@ -11,10 +11,10 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :github_repo_name: beats :discuss_forum: beats/{beatname_lc} :beat_default_index_prefix: {beatname_lc} -:has_ml_jobs: no :libbeat-docs: Beats Platform Reference :deb_os: :rpm_os: +:no_dashboards: include::../../libbeat/docs/shared-beats-attributes.asciidoc[] @@ -28,9 +28,6 @@ include::../../libbeat/docs/repositories.asciidoc[] include::./setting-up-running.asciidoc[] -//TODO: Decide whether this requires a separate topic -//include::./how-it-works.asciidoc[] - include::./configuring-howto.asciidoc[] include::./fields.asciidoc[] diff --git a/journalbeat/docs/overview.asciidoc b/journalbeat/docs/overview.asciidoc index 8ed7b455388..d8c018c38a4 100644 --- a/journalbeat/docs/overview.asciidoc +++ b/journalbeat/docs/overview.asciidoc @@ -5,8 +5,6 @@ Overview ++++ -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - {beatname_uc} is a lightweight shipper for forwarding and centralizing log data from https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html[systemd journals]. Installed as an agent on your servers, {beatname_uc} monitors the journal diff --git a/journalbeat/docs/running-on-kubernetes.asciidoc b/journalbeat/docs/running-on-kubernetes.asciidoc deleted file mode 100644 index 16b53e8e3af..00000000000 --- a/journalbeat/docs/running-on-kubernetes.asciidoc +++ /dev/null @@ -1,6 +0,0 @@ -[id="running-{beatname_lc}-on-kubernetes"] -=== Running {beatname_uc} on Kubernetes - -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - -ADD CONTENT HERE. diff --git a/journalbeat/docs/setting-up-running.asciidoc b/journalbeat/docs/setting-up-running.asciidoc index 8f7ccba3bb8..aeed49f8051 100644 --- a/journalbeat/docs/setting-up-running.asciidoc +++ b/journalbeat/docs/setting-up-running.asciidoc @@ -7,8 +7,6 @@ [[setting-up-and-running]] == Setting up and running {beatname_uc} -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - Before reading this section, see the <<{beatname_lc}-getting-started,getting started documentation>> for basic installation instructions to get you started. @@ -17,15 +15,8 @@ This section includes additional information on how to set up and run {beatname_uc}, including: * <> - * <> - * <> - -//* <> - -//* <> - * <> @@ -37,8 +28,4 @@ include::../../libbeat/docs/keystore.asciidoc[] include::../../libbeat/docs/command-reference.asciidoc[] -//include::./running-on-docker.asciidoc[] - -//include::./running-on-kubernetes.asciidoc[] - include::../../libbeat/docs/shared-shutdown.asciidoc[] diff --git a/journalbeat/docs/troubleshooting.asciidoc b/journalbeat/docs/troubleshooting.asciidoc index fa25622e7f1..3c14416e3dc 100644 --- a/journalbeat/docs/troubleshooting.asciidoc +++ b/journalbeat/docs/troubleshooting.asciidoc @@ -4,8 +4,6 @@ [partintro] -- -IMPORTANT: This documentation is placeholder content. It has not yet been reviewed. - If you have issues installing or running {beatname_uc}, read the following tips: diff --git a/libbeat/docs/command-reference.asciidoc b/libbeat/docs/command-reference.asciidoc index a6d666e78b7..928c3fbe047 100644 --- a/libbeat/docs/command-reference.asciidoc +++ b/libbeat/docs/command-reference.asciidoc @@ -14,7 +14,7 @@ :global-flags: Also see <>. -:export-command-short-desc: Exports the configuration, index template or a dashboard to stdout +:export-command-short-desc: Exports the configuration, index template, or a dashboard to stdout :help-command-short-desc: Shows help for any command :keystore-command-short-desc: Manages the <> :modules-command-short-desc: Manages configured modules @@ -97,7 +97,7 @@ Also see <>. {export-command-short-desc}. You can use this command to quickly view your configuration, see the contents of the index -template or export a dashboard from Kibana. +template, or export a dashboard from Kibana. *SYNOPSIS* @@ -439,10 +439,10 @@ endif::[] + ifdef::has_ml_jobs[] Loads the initial setup, including Elasticsearch template, Kibana index pattern, -Kibana dashboards and Machine learning jobs. +Kibana dashboards (when available), and Machine learning jobs. endif::[] ifndef::has_ml_jobs[] -Loads the initial setup, including Elasticsearch template, Kibana index pattern and Kibana dashboards. +Loads the initial setup, including Elasticsearch template, Kibana index pattern, and Kibana dashboards (when available). endif::[] If you want to use the command without running {beatname_uc}, use the <> command instead. @@ -511,10 +511,10 @@ environment without actually running {beatname_uc} and ingesting data. ifndef::deprecate_dashboard_loading[] *`--dashboards`*:: -Sets up the Kibana dashboards only. This option loads the dashboards from the -{beatname_uc} package. For more options, such as loading customized dashboards, -see {beatsdevguide}/import-dashboards.html[Importing Existing Beat Dashboards] -in the _Beats Developer Guide_. +Sets up the Kibana dashboards (when available). This option loads the dashboards +from the {beatname_uc} package. For more options, such as loading customized +dashboards, see {beatsdevguide}/import-dashboards.html[Importing Existing Beat +Dashboards] in the _Beats Developer Guide_. endif::[] ifdef::deprecate_dashboard_loading[] diff --git a/libbeat/docs/outputconfig.asciidoc b/libbeat/docs/outputconfig.asciidoc index fc949ed7979..ebd90e5cf96 100644 --- a/libbeat/docs/outputconfig.asciidoc +++ b/libbeat/docs/outputconfig.asciidoc @@ -215,9 +215,12 @@ The index name to write events to. The default is +"{beatname_lc}-%\{[beat.version]\}-%\{+yyyy.MM.dd\}"+ (for example, +"{beatname_lc}-{version}-{localdate}"+). If you change this setting, you also need to configure the `setup.template.name` and `setup.template.pattern` options -(see <>). If you are using the pre-built Kibana +(see <>). +ifndef::no_dashboards[] +If you are using the pre-built Kibana dashboards, you also need to set the `setup.dashboards.index` option (see <>). +endif::no_dashboards[] You can set the index dynamically by using a format string to access any event field. For example, this configuration uses a custom field, `fields.log_type`, diff --git a/libbeat/docs/security/securing-beats.asciidoc b/libbeat/docs/security/securing-beats.asciidoc index 29eff583df6..3d8ecda6175 100644 --- a/libbeat/docs/security/securing-beats.asciidoc +++ b/libbeat/docs/security/securing-beats.asciidoc @@ -67,12 +67,16 @@ ifeval::["{beatname_lc}"=="filebeat"] |Run Filebeat modules | ++{beat_default_index_prefix}_writer++ endif::[] |Load index templates | ++{beat_default_index_prefix}_writer++ and `kibana_user` +ifndef::no_dashboards[] |Load {beatname_uc} dashboards into {kib} | ++{beat_default_index_prefix}_writer++ and `kibana_user` +endif::[] ifdef::has_ml_jobs[] |Load machine learning jobs | `machine_learning_admin` endif::[] |Read indices created by {beatname_uc} | ++{beat_default_index_prefix}_reader++ +ifndef::no_dashboards[] |View {beatname_uc} dashboards in {kib} | `kibana_user` +endif::[] |======= include::basic-auth.asciidoc[] diff --git a/libbeat/docs/setup-config.asciidoc b/libbeat/docs/setup-config.asciidoc index 2e8b942c2e3..2fa6373f72c 100644 --- a/libbeat/docs/setup-config.asciidoc +++ b/libbeat/docs/setup-config.asciidoc @@ -1,4 +1,6 @@ - +ifndef::no_dashboards[] include::./dashboardsconfig.asciidoc[] +endif::no_dashboards[] + include::./template-config.asciidoc[] diff --git a/libbeat/docs/shared-configuring.asciidoc b/libbeat/docs/shared-configuring.asciidoc index c4008c4dc7f..52e2f9ad959 100644 --- a/libbeat/docs/shared-configuring.asciidoc +++ b/libbeat/docs/shared-configuring.asciidoc @@ -1,15 +1,8 @@ -ifndef::no-docker[] -To configure {beatname_uc}, you edit the configuration file. For rpm and deb, -you'll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+. Under -Docker, it's located at +/usr/share/{beatname_lc}/{beatname_lc}.yml+. For mac and win, -look in the archive that you just extracted. -endif::[] -ifdef::no-docker[] -To configure {beatname_uc}, you edit the configuration file. For rpm and deb, -you'll find the configuration file at +/etc/{beatname_lc}/{beatname_lc}.yml+. -For mac and win, look in the archive that you just extracted. -endif::[] +To configure {beatname_uc}, you edit the configuration file. The default +configuration file is called +{beatname_lc}.yml+. The location of the file +varies by platform. To locate the file, see <>. + ifeval::["{beatname_lc}"!="apm-server"] There’s also a full example configuration file called +{beatname_lc}.reference.yml+ that shows all non-deprecated options. diff --git a/libbeat/docs/shared-download-and-install.asciidoc b/libbeat/docs/shared-download-and-install.asciidoc index 0e3d608b9ec..0d434ba1650 100644 --- a/libbeat/docs/shared-download-and-install.asciidoc +++ b/libbeat/docs/shared-download-and-install.asciidoc @@ -2,19 +2,10 @@ *Before you begin*: If you haven't installed the {stack}, do that now. See {stack-gs}/get-started-elastic-stack.html[Getting started with the {stack}]. -ifndef::no-docker[] -To download and install {beatname_uc}, use the commands that work with your system -(<> for Debian/Ubuntu, <> for Redhat/Centos/Fedora, <> for OS X, <> for any Docker platform, and <> for -Windows). -endif::[] - -ifdef::no-docker[] -To download and install {beatname_uc}, use the commands that work with your system -(<> for Debian/Ubuntu, <> for Redhat/Centos/Fedora, <> for OS X, and <> for Windows). -endif::[] +To download and install {beatname_uc}, use the commands that work with your +system. +ifeval::["{release-state}"!="unreleased"] [NOTE] ================================================== @@ -24,3 +15,5 @@ repositories>> to update to the newest version more easily. See our https://www.elastic.co/downloads/beats/{beatname_lc}[download page] for other installation options, such as 32-bit images. ================================================== + +endif::[] diff --git a/libbeat/docs/shared-getting-started-intro.asciidoc b/libbeat/docs/shared-getting-started-intro.asciidoc index 84c6f07f1ae..ff2ca9dd6e5 100644 --- a/libbeat/docs/shared-getting-started-intro.asciidoc +++ b/libbeat/docs/shared-getting-started-intro.asciidoc @@ -7,7 +7,7 @@ related products: * {ls} (optional) for parsing and enhancing the data. See {stack-gs}/get-started-elastic-stack.html[Getting started with the {stack}] -for more information. +for more information about installing these products. [TIP] ============== diff --git a/libbeat/docs/shared-template-load.asciidoc b/libbeat/docs/shared-template-load.asciidoc index cd3670ad21a..f3d23d4d767 100644 --- a/libbeat/docs/shared-template-load.asciidoc +++ b/libbeat/docs/shared-template-load.asciidoc @@ -10,6 +10,11 @@ //// This content must be embedded underneath a level 3 heading. ////////////////////////////////////////////////////////////////////////// +ifndef::only-elasticsearch[] +NOTE: A connection to Elasticsearch is required to load the index template. If +the output is not Elasticsearch, you must +<>. +endif::only-elasticsearch[] In Elasticsearch, {elasticsearch}/indices-templates.html[index templates] are used to define settings and mappings that determine how fields @@ -21,35 +26,14 @@ The recommended index template file for {beatname_uc} is installed by the after successfully connecting to Elasticsearch. If the template already exists, it's not overwritten unless you configure {beatname_uc} to do so. -You can disable automatic template loading, or load your own template, by -configuring template loading options in the {beatname_uc} configuration file. - -You can also set options to change the name of the index and index template. - -ifndef::only-elasticsearch[] -NOTE: A connection to Elasticsearch is required to load the index template. If -the output is not Elasticsearch, you must -<>. -endif::only-elasticsearch[] - -For more information, see: - -ifdef::only-elasticsearch[] -* <> -* <> -endif::only-elasticsearch[] - -ifndef::only-elasticsearch[] -* <> -* <> - required for non-Elasticsearch output -endif::only-elasticsearch[] - [[load-template-auto]] ==== Configure template loading By default, {beatname_uc} automatically loads the recommended template file, -+fields.yml+, if the Elasticsearch output is enabled. You can change the -defaults in the +{beatname_lc}.yml+ config file to: ++fields.yml+, if the Elasticsearch output is enabled. If you want to use the +default index template, no additional configuration is required. Otherwise, you +can change the defaults in the +{beatname_lc}.yml+ config file +to: * **Load a different template** + @@ -89,35 +73,26 @@ that you specify should include the root name of the index plus version and date information. You also need to configure the `setup.template.name` and `setup.template.pattern` options to match the new name. For example: + -ifndef::deprecate_dashboard_loading[] - ["source","sh",subs="attributes,callouts"] ----- output.elasticsearch.index: "customname-%{[beat.version]}-%{+yyyy.MM.dd}" setup.template.name: "customname" setup.template.pattern: "customname-*" -setup.dashboards.index: "customname-*" <1> ----- - -<1> If you plan to -<>, also set -this option to overwrite the index name defined in the dashboards and index -pattern. - +ifndef::deprecate_dashboard_loading,no_dashboards[] ++ +If you're using pre-built Kibana dashboards, also set the +`setup.dashboards.index` option. For example: ++ +[source, yaml] +---- +setup.dashboards.index: "customname-*" +---- endif::[] ifdef::deprecate_dashboard_loading[] - -["source","sh",subs="attributes,callouts"] ------ -output.elasticsearch.index: "customname-%{[beat.version]}-%{+yyyy.MM.dd}" -setup.template.name: "customname" -setup.template.pattern: "customname-*" ------ + -Also ensure to change the index name accordingly in the Kibana dashboards, -when loading via the Kibana UI. - +Remember to change the index name when you load dashboards via the Kibana UI. endif::[] See <> for the full list of configuration options. diff --git a/libbeat/docs/step-configure-credentials.asciidoc b/libbeat/docs/step-configure-credentials.asciidoc index 6da1e51eb38..5b43e51fffe 100644 --- a/libbeat/docs/step-configure-credentials.asciidoc +++ b/libbeat/docs/step-configure-credentials.asciidoc @@ -30,8 +30,8 @@ values in the <>. <2> The `username` and `password` settings for {kib} are optional. If you don't specify credentials for {kib}, {beatname_uc} uses the `username` and `password` specified for the {es} output. -<3> If you are planning to <>, -the user must have the `kibana_user` -{xpack-ref}/built-in-roles.html[built-in role] or equivalent privileges. +<3> To use the pre-built Kibana dashboards, this user must have the +`kibana_user` {xpack-ref}/built-in-roles.html[built-in role] or equivalent +privileges. + For more information, see <>.